If retrieval based NLP [0] becomes a thing, then trillion plus parameters models will likely be less of a thing; as very likely, most of these tens to hundreds of billions of parameters are likely over-fitting (better word: memorized) on training data [text corpus] as seen in the case of GPT-3.
Yes, self-attention mechanisms are dense associative memories, so it might be possible to replace them in many cases with simpler storage mechanisms. Still, I would count the required storage space as part of a model's parameter size -- e.g., a model consisting of 1 trillion values in RAM and 99 trillion values in storage consists of... 100 trillion values.
Comments
If retrieval based NLP [0] becomes a thing, then trillion plus parameters models will likely be less of a thing; as very likely, most of these tens to hundreds of billions of parameters are likely over-fitting (better word: memorized) on training data [text corpus] as seen in the case of GPT-3.
[0] https://ai.stanford.edu/blog/retrieval-based-NLP/
Yes, self-attention mechanisms are dense associative memories, so it might be possible to replace them in many cases with simpler storage mechanisms. Still, I would count the required storage space as part of a model's parameter size -- e.g., a model consisting of 1 trillion values in RAM and 99 trillion values in storage consists of... 100 trillion values.