Comment on Training a 3.8B LLM to 0.384 CORE for $998parentComments−hadlock1dI've been training different architecture 1.33b and 3.33b models using nvidia megatron for about a month. You can train a 1.33b on a 46gb mac and 3.33b is about as big as you can go with a single 96gb blackwell with regular checkpoints etc
Comments
I've been training different architecture 1.33b and 3.33b models using nvidia megatron for about a month. You can train a 1.33b on a 46gb mac and 3.33b is about as big as you can go with a single 96gb blackwell with regular checkpoints etc