I am not sure how much experience you have in this area of research, but maybe I can shed some light on the background here. The "Attention is all you need" paper is now almost 7 years old. Those 7 years have seen a flood of proposals on improving transformers, only very few have been retained.
There is very little theoretic about transformer-style architectures. Fundamentally, the proof is in the pudding, not in "mathematical comparisons". A proposed change needs to scale better, it is all that matters. And the datasets mentioned are simply unsuitable for showing any scaling. I think the biggest dataset in this list, is 160MB compressed.
I am not sure why this article was posted here on hackernews. I would estimate even just today, there have probably been about 3 papers posted on arXiv with proposed transformer architecture changes, tested on larger datasets than the ones mentioned here.
I checked, and on the 28th of May, arXiv has seen 14 submissions with "transformer" in the title, and I found 3 of them with proposals tested on larger datasets (I did not check all of them, there might have been more than these three).
Comments
I am not sure how much experience you have in this area of research, but maybe I can shed some light on the background here. The "Attention is all you need" paper is now almost 7 years old. Those 7 years have seen a flood of proposals on improving transformers, only very few have been retained.
There is very little theoretic about transformer-style architectures. Fundamentally, the proof is in the pudding, not in "mathematical comparisons". A proposed change needs to scale better, it is all that matters. And the datasets mentioned are simply unsuitable for showing any scaling. I think the biggest dataset in this list, is 160MB compressed.
I am not sure why this article was posted here on hackernews. I would estimate even just today, there have probably been about 3 papers posted on arXiv with proposed transformer architecture changes, tested on larger datasets than the ones mentioned here.
I checked, and on the 28th of May, arXiv has seen 14 submissions with "transformer" in the title, and I found 3 of them with proposals tested on larger datasets (I did not check all of them, there might have been more than these three).
https://arxiv.org/pdf/2405.18240 https://arxiv.org/abs/2405.17951 https://arxiv.org/pdf/2405.17821
New is where progress comes from, so new is interesting. New is why we come here, and the first three letters of News.
Only way to fix that is with new.
"Can it scale" is something only someone with money can answer. It can be tested, but only if it's known. Now new is better known.