Comment on How has DeepSeek improved the Transformer architecture?parentComments−eldenring1yThey're not new in the same way Attention wasn't new when the transformer paper was written.No one (publically) had really pushed any of these techniques far, especially not for such a big run.−whimsicalism1yno one publicly pushes any techniques very far except for meta and it’s true they continue to train dense models for whatever reason.the transformer was an entirely new architecture, very different step change than thise: and alibaba−leetharris1yThey likely continue to train dense models because they are far easier to fine tune and this is a huge use case for the Llama models−whimsicalism1yIt probably also has to do with their internal infra. If it were just about dense models being easier for the OSS community to use & build on, they should probably be training MoEs and then distilling to dense.
Comments
They're not new in the same way Attention wasn't new when the transformer paper was written.
No one (publically) had really pushed any of these techniques far, especially not for such a big run.
no one publicly pushes any techniques very far except for meta and it’s true they continue to train dense models for whatever reason.
the transformer was an entirely new architecture, very different step change than this
e: and alibaba
They likely continue to train dense models because they are far easier to fine tune and this is a huge use case for the Llama models
It probably also has to do with their internal infra. If it were just about dense models being easier for the OSS community to use & build on, they should probably be training MoEs and then distilling to dense.