Base models are uniquely valuable as a research product because they accurately model the distribution of the data that they were trained on, as opposed to models that have had substantial RL tuning done to collapse their generation distributions. This makes base models the best starting point to fine-tune for a large number of different tasks.

Is this idea (‘collapse of their generation distributions’) a researched topic? If so, under what name?

Sounds interesting and maybe related to the whole continual learning / how to finetune properly line of work