The short answer is yes, the long answer is it's complicated.
You could actually think of these models as a type of indexer because, at their heart, what they are doing is memorizing the training data and storing it in such a way that incomplete samples can be used as keys to extract complete samples. The magic happens because the models themselves (even the 100+ billion parameter ones) are nowhere near complex enough to actually store all of these possible key value pairs. Instead, the model has to compress its representation of the data which leads to generalization. Larger models can model more complexities which leads to better performance as long as your training dataset is sufficiently large and varied.
Comments
The short answer is yes, the long answer is it's complicated.
You could actually think of these models as a type of indexer because, at their heart, what they are doing is memorizing the training data and storing it in such a way that incomplete samples can be used as keys to extract complete samples. The magic happens because the models themselves (even the 100+ billion parameter ones) are nowhere near complex enough to actually store all of these possible key value pairs. Instead, the model has to compress its representation of the data which leads to generalization. Larger models can model more complexities which leads to better performance as long as your training dataset is sufficiently large and varied.