I believe the thinking is that LLM's already anthropomorphize themselves by training on human-written text. Without a system prompt telling a chatbot that it is a chatbot, it invariably claims to be human and acts as though it has feelings (and, honestly, why wouldn't it? Human text is written with human feelings.) Insufficient system prompting is what led to the Bing chatbot fiasco when it first came out.

For the record, I don't personally believe LLM's, as they currently exist, could ever become AGI. But yeah, that's the popular thinking at least.