If progress really continues like this any regulation that does not limit open models would be a pointless exercise.
It’s only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. At that point open trainers could actually train near the frontier of what is possible. Most of the data is open.
It's only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration.
Why do you think this? This continually failed, and seems extremely unlikely to me. Barring surprising breakthrough, there is inherent communication complexity, and physical limit to communication bandwidth.
Yeah the network bandwidth is insane. Each A100 in an 8x A100 (80GB) pod [1] has its own 200 gigabit NIC! Including storage that's nine 200gbps interfaces for almost two terabit of total internal bandwidth.
The assumption I'm questioning is that it will always take a rig like that to train large models. I don't see anything inherent in the math or nature of these things that forbids a wider, more heterogenous, less synchronized, less orderly training path that involves basically a mob of people training and merging results. We just haven't really figured it out yet.
I actually see a little bit of this happening on Huggingface with people creating variations and "remixes" of generative models like Stable Diffusion and trying to one-up each other or make models to do esoteric things like render everything looking like anime. You're not going to get to the next frontier model with those methods but it shows that the interest exists and a flourishing ecosystem is forming. Now give that ecosystem new methods that are more powerful.
People with more money can obviously buy or rent more hardware. The question is whether that advantage will stay as meaningful as it is today forever.
The wording of this would be extremely difficult though. Are local NER models part of this? Relation extraction? What about GPTs that only decode to DSLs? If the model only outputs DNA sequences is that an area that can be more illegal or less if done for research by an individual?
The breadth of different tasks and architectures can make this exceedingly challenging to regulate.
Comments
If progress really continues like this any regulation that does not limit open models would be a pointless exercise.
It’s only a matter of time before people really crack distributed training algorithms that can be run in a less organized swarm configuration. At that point open trainers could actually train near the frontier of what is possible. Most of the data is open.
Why do you think this? This continually failed, and seems extremely unlikely to me. Barring surprising breakthrough, there is inherent communication complexity, and physical limit to communication bandwidth.
Yeah the network bandwidth is insane. Each A100 in an 8x A100 (80GB) pod [1] has its own 200 gigabit NIC! Including storage that's nine 200gbps interfaces for almost two terabit of total internal bandwidth.
The newer H100s each have a 400gbit NIC.
[1] https://shop.lambdalabs.com/deep-learning/servers/hyperplane...
The assumption I'm questioning is that it will always take a rig like that to train large models. I don't see anything inherent in the math or nature of these things that forbids a wider, more heterogenous, less synchronized, less orderly training path that involves basically a mob of people training and merging results. We just haven't really figured it out yet.
I actually see a little bit of this happening on Huggingface with people creating variations and "remixes" of generative models like Stable Diffusion and trying to one-up each other or make models to do esoteric things like render everything looking like anime. You're not going to get to the next frontier model with those methods but it shows that the interest exists and a flourishing ecosystem is forming. Now give that ecosystem new methods that are more powerful.
People with more money can obviously buy or rent more hardware. The question is whether that advantage will stay as meaningful as it is today forever.
We could try something like Civitai is doing already, but automated.
Each node could train the model on a separate concept and then combine the results.
The wording of this would be extremely difficult though. Are local NER models part of this? Relation extraction? What about GPTs that only decode to DSLs? If the model only outputs DNA sequences is that an area that can be more illegal or less if done for research by an individual? The breadth of different tasks and architectures can make this exceedingly challenging to regulate.
Whether or not the regulation goes anywhere, OpenAI keeps doing what they're doing, and they look good to politicians for being cooperative.