The trends around compute increasing is on the whole correct but by no means is it a universal rule. More optimized training procedures and model architectures are coming out all the time. In just the last week, we got F5-TTS which is trained on twice as much data as the previous leader in realistic TTS (Tortoise TTS) and is exponentially faster - taking only 3 weeks on H-100. We also got Meissonic, a text-to-image model that is exponentially easier to train than any existing model. IE you can train a Stable Diffusion like model from scratch on consumer hardware or in the cloud for abou $500.
The reason the trend is that compute costs are doubling is because this is an arms race and everyone in the corporate space is prioritizing bigger models over better architecture in the pursuit of a breakaway. It is not indicative of a law ala Moore's Law.
Comments
The trends around compute increasing is on the whole correct but by no means is it a universal rule. More optimized training procedures and model architectures are coming out all the time. In just the last week, we got F5-TTS which is trained on twice as much data as the previous leader in realistic TTS (Tortoise TTS) and is exponentially faster - taking only 3 weeks on H-100. We also got Meissonic, a text-to-image model that is exponentially easier to train than any existing model. IE you can train a Stable Diffusion like model from scratch on consumer hardware or in the cloud for abou $500.
https://huggingface.co/MeissonFlow/Meissonic
https://github.com/SWivid/F5-TTS
The reason the trend is that compute costs are doubling is because this is an arms race and everyone in the corporate space is prioritizing bigger models over better architecture in the pursuit of a breakaway. It is not indicative of a law ala Moore's Law.
Yes, to me, the report is interesting not because of the insights it offers, but because it raises more questions:
1. What about new AI chip vendors?
2. How will the price of compute change?
3. How will the demand for compute change?
4. How will the overall supply of chips change?