Comment on Megatron-Turing NLG 530B, the World’s Largest Generative Language ModelparentComments−canjobear4yParameters have been sufficient to memorize the training data for a while now. The fact that neural networks still generalize in this setting is a big mystery that is under active investigation.
Comments
Parameters have been sufficient to memorize the training data for a while now. The fact that neural networks still generalize in this setting is a big mystery that is under active investigation.