Absolutely, I have seen so many good ideas that have not yet made it into notable trained models.
A lot of that is because you need to have a lot more faith than "seems like a good idea" before you spend a few million in training that depends upon it.
Some of it is because when the models released now began training, a lot of those ideas hasn't been published yet.
Time will resolve most of that, cheaper and more performant hardware will allow a lot of those ideas to be tested without the massive commitment required to build the leading edge models.
The big guys are almost certainly incinerating millions a day on training "maybe it could show some promise" techniques. With the way things are right now, they are probably green lighting everything to find an edge.
Comments
I think DeepSeek (v3 and r1) showed us that there’s still a ton of meat on the bone for fundamental research and optimization.
Absolutely, I have seen so many good ideas that have not yet made it into notable trained models.
A lot of that is because you need to have a lot more faith than "seems like a good idea" before you spend a few million in training that depends upon it.
Some of it is because when the models released now began training, a lot of those ideas hasn't been published yet.
Time will resolve most of that, cheaper and more performant hardware will allow a lot of those ideas to be tested without the massive commitment required to build the leading edge models.
The big guys are almost certainly incinerating millions a day on training "maybe it could show some promise" techniques. With the way things are right now, they are probably green lighting everything to find an edge.