Yes, was my first reaction too, but then my 2nd reaction...
To put it as well as I can : after my 2nd reaction I went on to read the now available 2nd part, which also linked to this Open Letter by Yudkowski a month ago, which spells out my 2nd reaction probably better than I could :
I agree that current AIs are probably just imitating talk of self-awareness from their training data. But I mark that, with how little insight we have into these systems’ internals, we do not actually know.
Also, I am not convinced that our current research won't hit a brick wall well before super-intelligence, which would then require a radically different approach... but why are we willing to gamble with something as dangerous as this ?!
Don't we? Maybe someone with more architectural knowledge of LLM systems can fill in the gaps in my knowledge, but where's the feedback loop that would allow actual self-awareness to develop? FAFAIK, LLM's are feed-forward networks. There are neither small cycles spanning a few nodes, nor integral cycles allowing the model to see what it's outputting -- the only feedback it gets is through the training process, which does not act as a mirror.
I struggle to see how self-awareness would ever manifest from a system that cannot experience its own output.
Maybe someone with more architectural knowledge of LLM systems can fill in the gaps in my knowledge, but where’s the feedback loop that would allow actual self-awareness to develop? FAFAIK, LLM’s are feed-forward networks. There are neither small cycles spanning a few nodes, nor integral cycles allowing the model to see what it’s outputting – the only feedback it gets is through the training process, which does not act as a mirror.
This is, AFAIK, true of bare LLMs. Systems consisting of LLMs and a control interface that does reprompting which includes the earlier LLM responses (e.g., chat interfaces like ChatGPT) do provide the model with its output as feedback and show learning within the loop (though obviously this learning doesn’t reach back to the base model). The cognitive features of such a loop, such as they are, may well be different than those of a bare LLM considered in isolation.
I still think Yudkowsky is a crank and that taking him seriously is a bigger danger than AI.
I agree that we don't know exactly what is happening at any time in the model, but there's limits.
Sydney has no genitalia, no hormones, no sex drive. It does not "desire" someone. It may have a theory of mind, but clearly any AI saying that it loves someone is just working from its training data. The article treats this as though there was an actual intention there, rather than just a model repeating its training [0].
Just as obviously, this is a journo playing to our human desire to prevent aliens from running away with our women - any thought of non-humans breeding with humans triggers something pretty deep in us. Well played, sir. Take your paycheck, sit down, and please stop pushing the buttons.
The AIs are not going to take over the world in the sense that this article talks about. They are not invented by supernatural forces beyond our ken, as it suggests. This is a human emotional reaction to things that appear to be intelligent. I agree that this generation of AI is going to hit a brick wall and the shortcomings will become obvious, but the emotional reaction will be the same. I think it will be another 10-20 years before we solve some of the technical challenges in this generation of AI. But we're having the discussion about AI, and the emotional reaction to it, now. Which is probably good, because by the time we develop actual AI we'll have been living with our current level of "almost-AI" since we were kids and we'll be used to it.
And to answer the question: Every technology is dangerous. That has never stopped us from using it.
[0] Though, of course, that raises questions about what an "intention" is in human terms, too. Are we just "repeating our training"? I think not, at least for sex & love - there are too many hormones involved.
There is another technology that produced a similarly startling mystery -- the ouja board, introduced in 1890. That one, we figured out. The messages came from the depths of the human minds attached to the technology. The answers to the three numbered questions presented in the substack article are not known to me, but, of course, the nature of the presentation inclines me to think that the answers are likely to explain more about the operation of human minds when they attempt to collaborate in teams connected by unfathomed autonomous technology than they explain about the technology itself.
First sentence says:
The Internet and its consequences have been a disaster for the human race.
Root cause analysis gets me this far: The Internet is a consequence of the human race. The human race loves consequences. We cannot resist the allure of the search for advantageous consequences, even though we have declining brain size and increasing need for wisdom. It looks to me as if the fight for the wheel will include several factions of humans, several species of AI, and a jumble of alliances, hybrids, coalitions, strategies, conspiracies, and standoffs amongst them. The future is going to be just like a trip to the movies today -- nothing but sequels.
Does a spider desire to build a web? Maybe, maybe not, but it takes actions to try to build one. Is human desire a real thing, or is it an illusion? Some think that consciousness itself is an illusion. What is human sensation but the world sensing itself?
Does a country have desires? An economy? Do the millions of transactions and price changes and signals going between people and companies mean that an economy has thoughts, in some sense? When an economy reacts to a war in a foreign land like an ant flinching away from something hot, is it feeling pain in some very primitive sense?
Sure, it's entirely possible that there's no real sensation behind an LLM, or rather, it senses and reacts to text input like a bacterium reacting to chemical nutrient gradients or an ant reacting to food. But at what level of mental complexity does consciousness and true desire arise? I think it's a continuum, and to think that there's something special about biological intelligence vs machine intelligence is to believe in the supernatural. We wouldn't say that aliens don't have true desires because they have genetic material based on something other than DNA, or use something other than glucose for energy in their bodies.
I do think you're right that current LLMs proclaiming love likely don't understand what they're saying. But at some point we're going to build machine intelligences that do, and it may be hard to tell when we cross that line. People used to worry a lot more about the ethics of that.
Comments
Yes, was my first reaction too, but then my 2nd reaction...
To put it as well as I can : after my 2nd reaction I went on to read the now available 2nd part, which also linked to this Open Letter by Yudkowski a month ago, which spells out my 2nd reaction probably better than I could :
https://news.ycombinator.com/item?id=35364833
Also, I am not convinced that our current research won't hit a brick wall well before super-intelligence, which would then require a radically different approach... but why are we willing to gamble with something as dangerous as this ?!
we do not actually know
Don't we? Maybe someone with more architectural knowledge of LLM systems can fill in the gaps in my knowledge, but where's the feedback loop that would allow actual self-awareness to develop? FAFAIK, LLM's are feed-forward networks. There are neither small cycles spanning a few nodes, nor integral cycles allowing the model to see what it's outputting -- the only feedback it gets is through the training process, which does not act as a mirror.
I struggle to see how self-awareness would ever manifest from a system that cannot experience its own output.
This is, AFAIK, true of bare LLMs. Systems consisting of LLMs and a control interface that does reprompting which includes the earlier LLM responses (e.g., chat interfaces like ChatGPT) do provide the model with its output as feedback and show learning within the loop (though obviously this learning doesn’t reach back to the base model). The cognitive features of such a loop, such as they are, may well be different than those of a bare LLM considered in isolation.
I still think Yudkowsky is a crank and that taking him seriously is a bigger danger than AI.
I agree that we don't know exactly what is happening at any time in the model, but there's limits.
Sydney has no genitalia, no hormones, no sex drive. It does not "desire" someone. It may have a theory of mind, but clearly any AI saying that it loves someone is just working from its training data. The article treats this as though there was an actual intention there, rather than just a model repeating its training [0].
Just as obviously, this is a journo playing to our human desire to prevent aliens from running away with our women - any thought of non-humans breeding with humans triggers something pretty deep in us. Well played, sir. Take your paycheck, sit down, and please stop pushing the buttons.
The AIs are not going to take over the world in the sense that this article talks about. They are not invented by supernatural forces beyond our ken, as it suggests. This is a human emotional reaction to things that appear to be intelligent. I agree that this generation of AI is going to hit a brick wall and the shortcomings will become obvious, but the emotional reaction will be the same. I think it will be another 10-20 years before we solve some of the technical challenges in this generation of AI. But we're having the discussion about AI, and the emotional reaction to it, now. Which is probably good, because by the time we develop actual AI we'll have been living with our current level of "almost-AI" since we were kids and we'll be used to it.
And to answer the question: Every technology is dangerous. That has never stopped us from using it.
[0] Though, of course, that raises questions about what an "intention" is in human terms, too. Are we just "repeating our training"? I think not, at least for sex & love - there are too many hormones involved.
There is another technology that produced a similarly startling mystery -- the ouja board, introduced in 1890. That one, we figured out. The messages came from the depths of the human minds attached to the technology. The answers to the three numbered questions presented in the substack article are not known to me, but, of course, the nature of the presentation inclines me to think that the answers are likely to explain more about the operation of human minds when they attempt to collaborate in teams connected by unfathomed autonomous technology than they explain about the technology itself.
First sentence says:
Root cause analysis gets me this far: The Internet is a consequence of the human race. The human race loves consequences. We cannot resist the allure of the search for advantageous consequences, even though we have declining brain size and increasing need for wisdom. It looks to me as if the fight for the wheel will include several factions of humans, several species of AI, and a jumble of alliances, hybrids, coalitions, strategies, conspiracies, and standoffs amongst them. The future is going to be just like a trip to the movies today -- nothing but sequels.
Does a spider desire to build a web? Maybe, maybe not, but it takes actions to try to build one. Is human desire a real thing, or is it an illusion? Some think that consciousness itself is an illusion. What is human sensation but the world sensing itself?
Does a country have desires? An economy? Do the millions of transactions and price changes and signals going between people and companies mean that an economy has thoughts, in some sense? When an economy reacts to a war in a foreign land like an ant flinching away from something hot, is it feeling pain in some very primitive sense?
Sure, it's entirely possible that there's no real sensation behind an LLM, or rather, it senses and reacts to text input like a bacterium reacting to chemical nutrient gradients or an ant reacting to food. But at what level of mental complexity does consciousness and true desire arise? I think it's a continuum, and to think that there's something special about biological intelligence vs machine intelligence is to believe in the supernatural. We wouldn't say that aliens don't have true desires because they have genetic material based on something other than DNA, or use something other than glucose for energy in their bodies.
I do think you're right that current LLMs proclaiming love likely don't understand what they're saying. But at some point we're going to build machine intelligences that do, and it may be hard to tell when we cross that line. People used to worry a lot more about the ethics of that.