Autoregressive pretraining (text/images/video prediction) produces a foundational model. You can look at it as a highly compressed conditional probability distribution of the human brain output. The information-theoretically optimal compression of the data is a program that reproduces functionality of the process that generated the data.

So, it stands to reason (and observations) that such a model captures not only surface statistics of the data, but a part of functionality of the system that generated the data (the human brain for text, physics for video).