The assumption I'm questioning is that it will always take a rig like that to train large models. I don't see anything inherent in the math or nature of these things that forbids a wider, more heterogenous, less synchronized, less orderly training path that involves basically a mob of people training and merging results. We just haven't really figured it out yet.
I actually see a little bit of this happening on Huggingface with people creating variations and "remixes" of generative models like Stable Diffusion and trying to one-up each other or make models to do esoteric things like render everything looking like anime. You're not going to get to the next frontier model with those methods but it shows that the interest exists and a flourishing ecosystem is forming. Now give that ecosystem new methods that are more powerful.
People with more money can obviously buy or rent more hardware. The question is whether that advantage will stay as meaningful as it is today forever.
Comments
The assumption I'm questioning is that it will always take a rig like that to train large models. I don't see anything inherent in the math or nature of these things that forbids a wider, more heterogenous, less synchronized, less orderly training path that involves basically a mob of people training and merging results. We just haven't really figured it out yet.
I actually see a little bit of this happening on Huggingface with people creating variations and "remixes" of generative models like Stable Diffusion and trying to one-up each other or make models to do esoteric things like render everything looking like anime. You're not going to get to the next frontier model with those methods but it shows that the interest exists and a flourishing ecosystem is forming. Now give that ecosystem new methods that are more powerful.
People with more money can obviously buy or rent more hardware. The question is whether that advantage will stay as meaningful as it is today forever.