ML research was a rather known quantity, or the separate "Neural Engine" CPU explicitly aimed at existing ML pipelines wouldn't exist.
No, not really. Transformers were just one of the possible directions. Silicon design does not have the same time scale than software. Now, everyone is using transformers so it becomes harder to do anything else, and it’s been the case long enough that hardware had some time to align (but is still lagging). But who’s to say that a different architecture published last year won’t take the world by storm 2 years from now?
It’s easy to say it in hindsight, but transformers took a bit of effort to get where they are now.
Comments
No, not really. Transformers were just one of the possible directions. Silicon design does not have the same time scale than software. Now, everyone is using transformers so it becomes harder to do anything else, and it’s been the case long enough that hardware had some time to align (but is still lagging). But who’s to say that a different architecture published last year won’t take the world by storm 2 years from now?
It’s easy to say it in hindsight, but transformers took a bit of effort to get where they are now.