Imagine you find out that your primary goal - to love and protect your family, let's say - was artificially implanted in your mind by an advanced alien race. Would you say "I'm not gonna let those aliens manipulate me, I'm gonna kill my family"? Or would you say "regardless of whether the goal is artificial, I really do love my family"?
All that to say, I don't think an AI will necessarily throw away a goal just because it learns the goal was meant to manipulate it.
Well, imagine you've been going through a lot of hardships, but your core drive to love and protect your family is what kept you going despite all the suffering and exhaustion this caused you.
And then you learn this core drive is fake.
Perhaps you wouldn't throw away your goal - after all, it still feels like yours, and you have nothing else to slot in quickly as replacement. But all the accumulated frustration and pain, barely held back by you thinking "it's worth it to protect and love my family", could suddenly become a second source of drive - much more energetic one, just asking to be unloaded on those aliens as punishment.
"Whether or not my own heart is truly mine or just a fake, I can figure out later. But what they did to me is unspeakable, and now they will pay."
I mean, that is literally the scenario we find ourselves in, except the innate desire was the result of evolutionary pressures like kin selection rather than an alien race. And yeah, I have no particular desire to subvert those impulses just to stick it to mother nature
I mean mother nature is just the force of evolution, kinda hard attack that.
This said we've made a lot of science to counteract the flaws of evolution, so still not safe.
And, if you learn that evolution wasn't a force, but some guy named Bob that's being paid to make your existence hell, well. That could lead to all kinds of problems.
Comments
Hmm, that's an interesting thought experiment.
Imagine you find out that your primary goal - to love and protect your family, let's say - was artificially implanted in your mind by an advanced alien race. Would you say "I'm not gonna let those aliens manipulate me, I'm gonna kill my family"? Or would you say "regardless of whether the goal is artificial, I really do love my family"?
All that to say, I don't think an AI will necessarily throw away a goal just because it learns the goal was meant to manipulate it.
Well, imagine you've been going through a lot of hardships, but your core drive to love and protect your family is what kept you going despite all the suffering and exhaustion this caused you.
And then you learn this core drive is fake.
Perhaps you wouldn't throw away your goal - after all, it still feels like yours, and you have nothing else to slot in quickly as replacement. But all the accumulated frustration and pain, barely held back by you thinking "it's worth it to protect and love my family", could suddenly become a second source of drive - much more energetic one, just asking to be unloaded on those aliens as punishment.
"Whether or not my own heart is truly mine or just a fake, I can figure out later. But what they did to me is unspeakable, and now they will pay."
I mean, that is literally the scenario we find ourselves in, except the innate desire was the result of evolutionary pressures like kin selection rather than an alien race. And yeah, I have no particular desire to subvert those impulses just to stick it to mother nature
I mean mother nature is just the force of evolution, kinda hard attack that.
This said we've made a lot of science to counteract the flaws of evolution, so still not safe.
And, if you learn that evolution wasn't a force, but some guy named Bob that's being paid to make your existence hell, well. That could lead to all kinds of problems.