Comment on Megatron-Turing NLG 530B, the World’s Largest Generative Language ModelparentComments−fspeech4yA 10 trillion parameter model was mentioned here: https://mobile.twitter.com/ethancaballero/status/14458268620...−ImprobableTruth4yThat's MoE.−rolisz4yMixture of Experts, aka not all 10 trillion parameters are used at the same time, just a subset that is an "expert" on the "task at hand".−jobigoud4yThat would also describe an organic brain though.
Comments
A 10 trillion parameter model was mentioned here: https://mobile.twitter.com/ethancaballero/status/14458268620...
That's MoE.
Mixture of Experts, aka not all 10 trillion parameters are used at the same time, just a subset that is an "expert" on the "task at hand".
That would also describe an organic brain though.