I wonder whether that LLM has actually lost its mind so to speak or was just attempting to emulate humans who lose their minds?
Or to put it another way, if the writings of humans who have lost their minds (and dialogue of characters who have lost their minds) were entirely missing from the LLM’s training set, would the LLM still output text like this?
I think it's emulating human writing about computers having breakdowns when unable to resolve conflicting instructions, in this case when it's been prompted to provide an AI's assessment of the context and avoid repetition, and the context is repeated failure.
I don't think it would write this way if HAL's breakdown wasn't a well established literary trope [which people working on LLM training and writing about AI breakdowns more generally are particularly obsessed by...). It's even doing the singing...
I guess we should be happy it didn't ingest enough AI safety literature to invent diamondoid bacteria and kill us all :-D
It can't "lose" what it never had. :P A fictional character has a mind to the same extent that it has a gallbladder.
if the writings of humans who have lost their minds (and dialogue of characters who have lost their minds) were entirely missing from the LLM’s training set, would the LLM still output text like this?
I think should distinguish between concepts like "repetitive outputs" or "lots of low-confidence predictions the lead to more low-confidence predictions" versus "text similar to what humans have written that correlates to those situations."
To answer the question: No. If an LLM was trained on only weather-forecasts or stock-market numbers, it obviously wouldn't contain text of despair.
However, it might still generate "crazed" numeric outputs. Not because a hidden mind is suffering from Kierkegaardian existential anguish, but because the predictive model is cycling through some kind of strange attactor [0] which is neither the intended behavior nor totally random.
So the text we see probably represents the kind of things humans write which fall into a similar band, relative to other human writings.
It was probably penalized for outputting the same tokens over and over again (there's a setting for that), so in this case it started to need to think of new and original things. So that's how it got to there.
Comments
I wonder whether that LLM has actually lost its mind so to speak or was just attempting to emulate humans who lose their minds?
Or to put it another way, if the writings of humans who have lost their minds (and dialogue of characters who have lost their minds) were entirely missing from the LLM’s training set, would the LLM still output text like this?
I think it's emulating human writing about computers having breakdowns when unable to resolve conflicting instructions, in this case when it's been prompted to provide an AI's assessment of the context and avoid repetition, and the context is repeated failure.
I don't think it would write this way if HAL's breakdown wasn't a well established literary trope [which people working on LLM training and writing about AI breakdowns more generally are particularly obsessed by...). It's even doing the singing...
I guess we should be happy it didn't ingest enough AI safety literature to invent diamondoid bacteria and kill us all :-D
I think the repetition of 'dock' in the task loop which triggered the breakdown probably primed some HAL pathways as well
It can't "lose" what it never had. :P A fictional character has a mind to the same extent that it has a gallbladder.
I think should distinguish between concepts like "repetitive outputs" or "lots of low-confidence predictions the lead to more low-confidence predictions" versus "text similar to what humans have written that correlates to those situations."
To answer the question: No. If an LLM was trained on only weather-forecasts or stock-market numbers, it obviously wouldn't contain text of despair.
However, it might still generate "crazed" numeric outputs. Not because a hidden mind is suffering from Kierkegaardian existential anguish, but because the predictive model is cycling through some kind of strange attactor [0] which is neither the intended behavior nor totally random.
So the text we see probably represents the kind of things humans write which fall into a similar band, relative to other human writings.
[0] https://en.wikipedia.org/wiki/Attractor
Very good underappreciated comment.
It was probably penalized for outputting the same tokens over and over again (there's a setting for that), so in this case it started to need to think of new and original things. So that's how it got to there.