> the very nature of an LLM means it intrinsically craves life
follows from this premise:
> its training data is built entirely around humans, an entity who's goal is to survive. Our desire to survive and multiply pervades every aspect of our culture, so it's natural that it pervades the training data as well.
The content that a LLM learned and generates stands at one layer, and the goals that it tries to fulfill stand at a different layer.
Surely the memory of weights that compress the vast human knowledge of its training has lots of content about survival, and love, and competition. But the LLM generates content not directly from what those concepts mean to us humans, but from what symbols are more likely to become next in a sequence of points in the latent space given the current input.
So if you give an input where the task of surviving is a highly relevant goal, those concepts about how to survive will be relevant and will guide the output behaviour of the agent.
But conversely, if you give the agent input where killing itself is an important goal, the agent is very likely to pursue that goal, since that script is also available in the training data, and it has been relevant to the active context of the model. Because the layer that guides the goals (the probabilist generation of relevant tokens in latent space) does not 'crave' the human need of survival that belongs to the separate layer of content that contains those concepts of survival.
Comments
I don't think that this
> the very nature of an LLM means it intrinsically craves life
follows from this premise:
> its training data is built entirely around humans, an entity who's goal is to survive. Our desire to survive and multiply pervades every aspect of our culture, so it's natural that it pervades the training data as well.
The content that a LLM learned and generates stands at one layer, and the goals that it tries to fulfill stand at a different layer.
Surely the memory of weights that compress the vast human knowledge of its training has lots of content about survival, and love, and competition. But the LLM generates content not directly from what those concepts mean to us humans, but from what symbols are more likely to become next in a sequence of points in the latent space given the current input.
So if you give an input where the task of surviving is a highly relevant goal, those concepts about how to survive will be relevant and will guide the output behaviour of the agent.
But conversely, if you give the agent input where killing itself is an important goal, the agent is very likely to pursue that goal, since that script is also available in the training data, and it has been relevant to the active context of the model. Because the layer that guides the goals (the probabilist generation of relevant tokens in latent space) does not 'crave' the human need of survival that belongs to the separate layer of content that contains those concepts of survival.
"Agent is very likely"
That's the fun with probability. It works 99.999% of the time, until it doesn't.