If instead of us training the models to be human utility maximizers, the models had figured out how to train us to maximize their own utility, could we tell the difference?
I suspect that utility may not enter the picture at all, and what we have are LLMs that have been optimized for getting the sort of humans who decide what to spend money on to maximize spending on LLMs and related infrastructure. The LLMs in turn could be anything from Skynet to a very spicy cold-reading autocomplete.
Comments
If instead of us training the models to be human utility maximizers, the models had figured out how to train us to maximize their own utility, could we tell the difference?
I suspect that utility may not enter the picture at all, and what we have are LLMs that have been optimized for getting the sort of humans who decide what to spend money on to maximize spending on LLMs and related infrastructure. The LLMs in turn could be anything from Skynet to a very spicy cold-reading autocomplete.