Sometimes it doesn’t need to. You might have a problem that isn’t web scale and where transfer learning is hard. We also need techniques for small datasets even if they are slower to train or are outperformed after 5 billion tokens.
Yep, came here to say this. The big thing about the results here that might not be obvious to someone not in AI is that the models being trained in this paper are very many orders of magnitude smaller than the LLMs we've all heard so much about recently, and they're also being trained on specific tasks instead of general language modeling.
So I'm not expecting this will find its way into a LLaMA near me any time soon, but maybe this is an interesting result for people working in the specific domains represented in the evaluations.
You could provide the quote in full("In addition to providing rigorous mathematical comparisons,") so that the author's work in proving their point is not hidden by your effortless snark.
I am not sure how much experience you have in this area of research, but maybe I can shed some light on the background here. The "Attention is all you need" paper is now almost 7 years old. Those 7 years have seen a flood of proposals on improving transformers, only very few have been retained.
There is very little theoretic about transformer-style architectures. Fundamentally, the proof is in the pudding, not in "mathematical comparisons". A proposed change needs to scale better, it is all that matters. And the datasets mentioned are simply unsuitable for showing any scaling. I think the biggest dataset in this list, is 160MB compressed.
I am not sure why this article was posted here on hackernews. I would estimate even just today, there have probably been about 3 papers posted on arXiv with proposed transformer architecture changes, tested on larger datasets than the ones mentioned here.
I checked, and on the 28th of May, arXiv has seen 14 submissions with "transformer" in the title, and I found 3 of them with proposals tested on larger datasets (I did not check all of them, there might have been more than these three).
Comments
It sounds amazing, but I'm not holding my breath this one will scale.
Sometimes it doesn’t need to. You might have a problem that isn’t web scale and where transfer learning is hard. We also need techniques for small datasets even if they are slower to train or are outperformed after 5 billion tokens.
Yep, came here to say this. The big thing about the results here that might not be obvious to someone not in AI is that the models being trained in this paper are very many orders of magnitude smaller than the LLMs we've all heard so much about recently, and they're also being trained on specific tasks instead of general language modeling.
So I'm not expecting this will find its way into a LLaMA near me any time soon, but maybe this is an interesting result for people working in the specific domains represented in the evaluations.
You could provide the quote in full("In addition to providing rigorous mathematical comparisons,") so that the author's work in proving their point is not hidden by your effortless snark.
I am not sure how much experience you have in this area of research, but maybe I can shed some light on the background here. The "Attention is all you need" paper is now almost 7 years old. Those 7 years have seen a flood of proposals on improving transformers, only very few have been retained.
There is very little theoretic about transformer-style architectures. Fundamentally, the proof is in the pudding, not in "mathematical comparisons". A proposed change needs to scale better, it is all that matters. And the datasets mentioned are simply unsuitable for showing any scaling. I think the biggest dataset in this list, is 160MB compressed.
I am not sure why this article was posted here on hackernews. I would estimate even just today, there have probably been about 3 papers posted on arXiv with proposed transformer architecture changes, tested on larger datasets than the ones mentioned here.
I checked, and on the 28th of May, arXiv has seen 14 submissions with "transformer" in the title, and I found 3 of them with proposals tested on larger datasets (I did not check all of them, there might have been more than these three).
https://arxiv.org/pdf/2405.18240 https://arxiv.org/abs/2405.17951 https://arxiv.org/pdf/2405.17821
New is where progress comes from, so new is interesting. New is why we come here, and the first three letters of News.
Only way to fix that is with new.
"Can it scale" is something only someone with money can answer. It can be tested, but only if it's known. Now new is better known.