Maybe someone with more architectural knowledge of LLM systems can fill in the gaps in my knowledge, but where’s the feedback loop that would allow actual self-awareness to develop? FAFAIK, LLM’s are feed-forward networks. There are neither small cycles spanning a few nodes, nor integral cycles allowing the model to see what it’s outputting – the only feedback it gets is through the training process, which does not act as a mirror.

This is, AFAIK, true of bare LLMs. Systems consisting of LLMs and a control interface that does reprompting which includes the earlier LLM responses (e.g., chat interfaces like ChatGPT) do provide the model with its output as feedback and show learning within the loop (though obviously this learning doesn’t reach back to the base model). The cognitive features of such a loop, such as they are, may well be different than those of a bare LLM considered in isolation.

I still think Yudkowsky is a crank and that taking him seriously is a bigger danger than AI.