Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.
Parameters have been sufficient to memorize the training data for a while now. The fact that neural networks still generalize in this setting is a big mystery that is under active investigation.
I've often wondered if a lighter reinforcement learning based model on top of a full text index might do as well or better than these putatively overfit language models. Curious if anyone knows of ongoing or recent work on this approach.
Comments
Training data has 0.339T tokens, less than the number of training parameters. A model like that could store all of the training text with 100B+ parameters left for computation.
Parameters have been sufficient to memorize the training data for a while now. The fact that neural networks still generalize in this setting is a big mystery that is under active investigation.
For some reason this issue with model having insane amounts of weights but training data being small is not something that is an issue for modern NNs.
https://arxiv.org/abs/2109.02355
But then you try to predict the next token on a completely unseen piece of the corpus and fail miserably if all you do is store the training data.
A single weight can’t encode an individual word, but the ratio looks close to overfitting too me too.
I've often wondered if a lighter reinforcement learning based model on top of a full text index might do as well or better than these putatively overfit language models. Curious if anyone knows of ongoing or recent work on this approach.
If 16-bit floating-point numbers are used, it can presumably encode all tokens. In theory. It would not be very easy to work with.
Maybe that’s what it’s doing under the hood.