I hope we will soon have an open-source project for training such small LLMs where one can just pick the architecture (like Qwen / DeepSeek / etc.), parameter count, dataset, ... and then let it run on a local/rented GPUs...
I’m with you. But what are we going to get? I think this goal sits in a funny place between knowledge and convenience.
On the one hand, tons of “products” promising this.
On the other hand, I’m sure we can find student works—sharing the code they created from a course or book. And I expect there will be gaps, niches filled by pro-coders who see a pro need and fill it.
What I don’t want to do is see the market for model training filled with whatever Microsoft thinks will make money.
I've been training different architecture 1.33b and 3.33b models using nvidia megatron for about a month. You can train a 1.33b on a 46gb mac and 3.33b is about as big as you can go with a single 96gb blackwell with regular checkpoints etc
Comments
I hope we will soon have an open-source project for training such small LLMs where one can just pick the architecture (like Qwen / DeepSeek / etc.), parameter count, dataset, ... and then let it run on a local/rented GPUs...
I’m with you. But what are we going to get? I think this goal sits in a funny place between knowledge and convenience.
On the one hand, tons of “products” promising this.
On the other hand, I’m sure we can find student works—sharing the code they created from a course or book. And I expect there will be gaps, niches filled by pro-coders who see a pro need and fill it.
What I don’t want to do is see the market for model training filled with whatever Microsoft thinks will make money.
I've been training different architecture 1.33b and 3.33b models using nvidia megatron for about a month. You can train a 1.33b on a 46gb mac and 3.33b is about as big as you can go with a single 96gb blackwell with regular checkpoints etc