No, that's what you're missing from the beginning: the breakthrough of transformers was scalability. Now we have other models that are equally scalable and as such roughly equally performant (and that's not a surprise).
But the ship has sailed and nobody is gonna switch to something else than transformers if it's not significantly better, and as such the other approaches are going to stay behind because every marginal improvement come to transformers first (because that's what practically everyone is working on) and alternative models are playing catch-up.
This is a remarkable example of path dependence.
Interpreting this as “transformers are fundamentally superior” is the mistake I'm trying to help you correct.
The breakthrough of transformers was scalability. The next breakthrough of equivalent importance will be entirely different or it won't be.
Comments
Yes, a breakthrough that does what Self-Attention is doing, rather than just scaling up.
No, that's what you're missing from the beginning: the breakthrough of transformers was scalability. Now we have other models that are equally scalable and as such roughly equally performant (and that's not a surprise).
But the ship has sailed and nobody is gonna switch to something else than transformers if it's not significantly better, and as such the other approaches are going to stay behind because every marginal improvement come to transformers first (because that's what practically everyone is working on) and alternative models are playing catch-up.
This is a remarkable example of path dependence.
Interpreting this as “transformers are fundamentally superior” is the mistake I'm trying to help you correct.
The breakthrough of transformers was scalability. The next breakthrough of equivalent importance will be entirely different or it won't be.
Scalability wasn't an architectural breakthrough. It was merely a discovery..
How are these words even in contradiction to each other?
By intentionally lacking context.
You aren't even trying to pretend your sentence make sense, I see…
so Kindergarten bro.
We are now at the Markov chain level of sentence, it's not even grammatically resembling a sentence at that point. Congratz!
Like your 1 word replies.
No.
Your post attacking my grammar contains two grammatical errors of it's own. Can you spot them?
I'm not attacking your grammar, but now I question your ability to understand a sentence.
(English isn't my first language, BTW, so I'd be grateful if you could point the grammar errors)
The 4 words that inadvertently summarized your entire disposition.
It's not inadvertent.
You've spent all your credit for any form of consideration at that point.
And now you've even spent your credit for my attention at all, this conversation ends here.
By "inadvertently" I meant you didn't intend the 4 words as an encapsulation of your entire persona.