It may not be a satisfying answer, but it is a correct one.
The comparison to the underpinnings of human cognition are not really valid, because regardless of the underlying substrate, human beings are indeed building mental models of reality itself via sensory input, so even if human minds and LLMs operated on the exact same process of inference (which we cannot state with confidence is true), then human minds would be generating inferences based on correlations of actual empirical experience, whereas LLMs are only building correlations between words, regardless of what empirical reality those words may or may not correspond to.
So all else being equal, human minds are modeling reality, while LLMs are modeling another model.
Comments
It may not be a satisfying answer, but it is a correct one.
The comparison to the underpinnings of human cognition are not really valid, because regardless of the underlying substrate, human beings are indeed building mental models of reality itself via sensory input, so even if human minds and LLMs operated on the exact same process of inference (which we cannot state with confidence is true), then human minds would be generating inferences based on correlations of actual empirical experience, whereas LLMs are only building correlations between words, regardless of what empirical reality those words may or may not correspond to.
So all else being equal, human minds are modeling reality, while LLMs are modeling another model.