Comment on Megatron-Turing NLG 530B, the World’s Largest Generative Language ModelparentComments−ImprobableTruth4yThat's MoE.−rolisz4yMixture of Experts, aka not all 10 trillion parameters are used at the same time, just a subset that is an "expert" on the "task at hand".−jobigoud4yThat would also describe an organic brain though.
Comments
That's MoE.
Mixture of Experts, aka not all 10 trillion parameters are used at the same time, just a subset that is an "expert" on the "task at hand".
That would also describe an organic brain though.