It's only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration.
Why do you think this? This continually failed, and seems extremely unlikely to me. Barring surprising breakthrough, there is inherent communication complexity, and physical limit to communication bandwidth.
Yeah the network bandwidth is insane. Each A100 in an 8x A100 (80GB) pod [1] has its own 200 gigabit NIC! Including storage that's nine 200gbps interfaces for almost two terabit of total internal bandwidth.
The assumption I'm questioning is that it will always take a rig like that to train large models. I don't see anything inherent in the math or nature of these things that forbids a wider, more heterogenous, less synchronized, less orderly training path that involves basically a mob of people training and merging results. We just haven't really figured it out yet.
I actually see a little bit of this happening on Huggingface with people creating variations and "remixes" of generative models like Stable Diffusion and trying to one-up each other or make models to do esoteric things like render everything looking like anime. You're not going to get to the next frontier model with those methods but it shows that the interest exists and a flourishing ecosystem is forming. Now give that ecosystem new methods that are more powerful.
People with more money can obviously buy or rent more hardware. The question is whether that advantage will stay as meaningful as it is today forever.
Comments
Why do you think this? This continually failed, and seems extremely unlikely to me. Barring surprising breakthrough, there is inherent communication complexity, and physical limit to communication bandwidth.
Yeah the network bandwidth is insane. Each A100 in an 8x A100 (80GB) pod [1] has its own 200 gigabit NIC! Including storage that's nine 200gbps interfaces for almost two terabit of total internal bandwidth.
The newer H100s each have a 400gbit NIC.
[1] https://shop.lambdalabs.com/deep-learning/servers/hyperplane...
The assumption I'm questioning is that it will always take a rig like that to train large models. I don't see anything inherent in the math or nature of these things that forbids a wider, more heterogenous, less synchronized, less orderly training path that involves basically a mob of people training and merging results. We just haven't really figured it out yet.
I actually see a little bit of this happening on Huggingface with people creating variations and "remixes" of generative models like Stable Diffusion and trying to one-up each other or make models to do esoteric things like render everything looking like anime. You're not going to get to the next frontier model with those methods but it shows that the interest exists and a flourishing ecosystem is forming. Now give that ecosystem new methods that are more powerful.
People with more money can obviously buy or rent more hardware. The question is whether that advantage will stay as meaningful as it is today forever.
We could try something like Civitai is doing already, but automated.
Each node could train the model on a separate concept and then combine the results.