We can't even track nuclear programs all that accurately and those create a radiological and isotope signature that can often be detected from space. We couldn't even stop a relatively small nation like North Korea from developing and testing nukes.
This proposal is just so clearly unenforceable and infeasible. Attempts to pause scientific progress have never worked in the past. How would it ever work now, especially when the cryptobros around the world have already been amassing hidden GPU clusters for mining.
Robert Miles (AI risk YouTuber) made some good points, which is that we've made effective agreements to stop things like human cloning and weapons that blind people with lasers. It can be done. We did it with things like CFCs as well.
Even if we can turn AI research into a criminal activity would be a huge step in the right direction. At the moment, it's a matter of buying time.
I'm not wearing rose-tinted glasses about the difficulty. I'm already trying to work though the grieving process, myself. But I think it's worth putting in the effort, even if the end result is just reducing the chance of doom by a fraction of a percentage point, or buying humanity 2 or 3 more years.
Banning blinding laser weapons was easy to get consensus on. Banning AI research is more like banning laser research.
I have read Bostrom's Superintelligence: Paths, Dangers, Strategies so I think I have reasonable exposure to the arguments that AI could drive humans extinct. But I didn't find any one of the scenarios plausible enough to frighten me. If there is a new strongest argument for why AI is too dangerous, developed since that 2014 book, I'm willing to read it.
Yudkowsky's open letter makes me think that the arguments are not a lot stronger now than in 2014:
A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing.
Doing anything novel with biology requires experimentation. A million-times-faster thinker won't be able to advance biological research a million times faster. It'll be able to do the experimental design parts faster, and the post-experimental evaluation faster, but it'll have to wait just as long as a human postdoc waits for things to grow and assays to run. It's Amdahl's Law applied to scientific research: the achievable speedup is a modest multiple because even unsleeping geniuses have to wait for experiments.
If I were to interpret Yudkowsky's "bootstrap straight to postbiological molecular manufacturing" in the most charitable way possible, maybe he's using an implausible scenario as a sort of didactic scary story when trying to communicate his fears to the public. But I'd then like to understand and evaluate the actual scenario he's talking about.
Doing something significant with biology requires experimentation for humans. That doesn’t mean it does for every intelligence. It’s quite possible that we have collected enough data in every scientific paper ever written (which an AI could process and understand instantaneously) that an accurate simulation could be run within a computing context. If there are parts of the simulation that are inaccurate, the AI could design physical experiments that are more controlled and isolated to answer any unknowns and make the simulation more accurate with minimal iterations. Assuming this AI is massively more intelligent than us, it isn’t unfeasible that it could find some zero-day exploits to take over all cloud computing to run the simulations, for example, so even the compute unavailability isn’t a great argument as to why it couldn’t do this.
Intelligence is a limited, partial substitute for knowledge. The problem with Aristotelian physics wasn't that Aristotle wasn't smart enough. It was that Aristotle lacked the ground truth knowledge to understand where he was wrong. An AI that can't run experiments will end up with elegant hallucinations of nanofactory blueprints but no actual nanofactory. (That's even assuming that Drexlerian nanotech is actually something that can be built. It's still an open question.)
All the computing power in the world is insufficient to simulate a living bacterium at the molecular level. We also don't have a molecular resolution map of bacteria in the first place. The AI will have to figure out how to make one. The AI can try to develop faster high fidelity approximations for molecular simulation, but that too requires calibration against experiments as well as a lot of simulation resources. If it seizes control of public cloud computing for its simulations people will notice and can just unplug cables. If its intrusions are subtle, only using little unnoticeable compute slices when machines are largely idle, then its simulations are going to get less compute resources in aggregate than researchers have now.
You're right about laser weapons vs laser research. A real problem is that non-lethal AI is very lucrative.
I haven't read the book, so I'm not familiar with which scenarios in particular Bostrom talks about. But I believe I'm versed enough in the issues with AI alignment to carry on an intelligent conversation about it. When you say that none of the scenarios weren't plausible enough, could you give an example as to what you mean?
To give an example from my end, I'm really just imagining a scenario where we give the AGI some sort of goal, such as curing cancer. We've thought about the ways things could go wrong, and used reinforcement learning from human feedback to make an AI that seems to have learned not to do anything that could cause serious harm to a human in training. But it turns out that it actually learned to game our rewards, and just learned not to do anything to harm humans that could be traced back to it. Similarly, we tried to train it not to acquire more resources and only use what we give it, but it just learned to hide its activity. We know reward hacking is extremely common in AIs trained this way. GPT-4 is still willing to provide users with forbidden information -- it hasn't actually internalized the goal of "don't provide people with information that they could likely use to harm themselves or others", it's just some superficially similar goal that we don't really understand.
So anyway, the AI didn't actually learn to care about humanity, only about putting on that appearance, like a sociopath that learned to fake emotions to blend in. It has learned the ability to reason and plan, in order to design virtual experiments to calculate protein folding, etc. We provide it with a plain language description of the problem we want solved, specifying a ton of caveats like:
* Don't take actions that would result in the greater chance of death of people with cancer.
* Don't try to detect people with a high likelihood of cancer and kill people before they get it.
* Try to come up with a solution to this that most people would be happy with.
We can't specify things like "don't do anything that might result in someone dying" because in previous tests the AI would refuse to try to cure cancer, in case someone who was cured then went on to kill someone else.
No matter how much we specify, it turns out that a sufficiently intelligent, creative AI can come up with a maximal solution. Say, it secretly creates a drug that it distributes through the water supply that makes most people happy with just about anything, then builds a bunch of suspended animation chambers where it stores people who have cancer for ever -- they can be revived, so they're not actually dead. It needs to keep things running forever, so of course it has to take over things in order to build enough of these chambers for everyone who ever lived. Now, that's not really possible, if people keep breeding and reproducing, so it ends up just sticking everyone into these chambers, which they're fine with because of the happy-drug. The earth continues silently through space, with all of humanity protected in suspended animation by our over-zealous AI. Oh, and any life that we mentioned in restrictions is also in suspended animation, while every other bit of life has been exterminated as the AI tried to maximize its resources in order to keep this state of affairs in place forever.
It's a thing where no matter how many restrictions you try to stick on to the problem, a sufficiently smart AI can probably come up with a better solution in one of the loopholes than it can by just doing the boring, human thing of trying to figure out an effective cancer drug that works in 98% of cases. The AI tries to get to 100%, which necessitates a more drastic solution. If you tell it just to come up with a drug with a greater than 90% effectiveness rate, it STILL wants to take over the world in order to acquire enough computing power to guarantee that the effectiveness is no less than 90%. We don't know how to build "satisficers" that are happy with good enough, and even if we could, "satisficers" will tend to build maximizers.
That's exactly what humans (which are kinda satisficers, sometimes) are doing by building AI -- we're trying to find maximal solutions to our problems, except I'm worried that we're not quite smart enough so we're gonna screw it up real bad.
I think we have a chance of building an AI that's aligned close enough not to kill us, as long as it's not that much smarter than us. But the smarter it gets, the more that 2-degree of misalignment matters. And they're already terrifyingly smart.
The AI doomism will help no one. Bing search is a good example. Very good simulation of an average human. Sometime same as hysterical as humans online...
The thing is it is a simulation. The AI however smart it will be, will miss something very important for someone to want to eradicate humanity. It will miss the "will" and the erratic harmonal storms of our body and brain. It will miss the drive to want something. On the other hand, a good simulation of an average or even psychotic human on an AI capable to make this simulation smarter then all other humans, can be dangerous to humanity. Thus: STOP FEEDING AI with our biases and behavioral data, stop feeding it with the chat data and the literature data and the tones of online trash which is posted every day all around the internet. Stop feeding it with the data which will make it possible to simulate an average human. 32K context and possibility to focus on 3-4 key-points of the conversation now, will soon (like 2 years from now) change and will turn into 64k and 7-8 key-points. This is still not enough to build a long term strategy to eradicate humanity.
The AI however smart it will be, will miss something very important for someone to want to eradicate humanity. It will miss the "will" and the erratic harmonal storms of our body and brain. It will miss the drive to want something.
AI wouldn't need that to win at scale, I don't think it would play Drunken Kung Fu Master against humanity but rather perform millions of time efficient, calculated movements a'la StarCraft. Otherwise I agree these are traits that are beneficial for us to fight back. Don't mind me, just speculating.
Humanity got on the top of the food chain, not only because it is smarter and can turn predators into prey. Humans have a will to do so. Even animals have a will and biological needs. AI can't have it unless it simulates it. That's about it.
Comments
We can't even track nuclear programs all that accurately and those create a radiological and isotope signature that can often be detected from space. We couldn't even stop a relatively small nation like North Korea from developing and testing nukes.
This proposal is just so clearly unenforceable and infeasible. Attempts to pause scientific progress have never worked in the past. How would it ever work now, especially when the cryptobros around the world have already been amassing hidden GPU clusters for mining.
Robert Miles (AI risk YouTuber) made some good points, which is that we've made effective agreements to stop things like human cloning and weapons that blind people with lasers. It can be done. We did it with things like CFCs as well.
Even if we can turn AI research into a criminal activity would be a huge step in the right direction. At the moment, it's a matter of buying time.
I'm not wearing rose-tinted glasses about the difficulty. I'm already trying to work though the grieving process, myself. But I think it's worth putting in the effort, even if the end result is just reducing the chance of doom by a fraction of a percentage point, or buying humanity 2 or 3 more years.
Banning blinding laser weapons was easy to get consensus on. Banning AI research is more like banning laser research.
I have read Bostrom's Superintelligence: Paths, Dangers, Strategies so I think I have reasonable exposure to the arguments that AI could drive humans extinct. But I didn't find any one of the scenarios plausible enough to frighten me. If there is a new strongest argument for why AI is too dangerous, developed since that 2014 book, I'm willing to read it.
Yudkowsky's open letter makes me think that the arguments are not a lot stronger now than in 2014:
A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing.
Doing anything novel with biology requires experimentation. A million-times-faster thinker won't be able to advance biological research a million times faster. It'll be able to do the experimental design parts faster, and the post-experimental evaluation faster, but it'll have to wait just as long as a human postdoc waits for things to grow and assays to run. It's Amdahl's Law applied to scientific research: the achievable speedup is a modest multiple because even unsleeping geniuses have to wait for experiments.
If I were to interpret Yudkowsky's "bootstrap straight to postbiological molecular manufacturing" in the most charitable way possible, maybe he's using an implausible scenario as a sort of didactic scary story when trying to communicate his fears to the public. But I'd then like to understand and evaluate the actual scenario he's talking about.
Doing something significant with biology requires experimentation for humans. That doesn’t mean it does for every intelligence. It’s quite possible that we have collected enough data in every scientific paper ever written (which an AI could process and understand instantaneously) that an accurate simulation could be run within a computing context. If there are parts of the simulation that are inaccurate, the AI could design physical experiments that are more controlled and isolated to answer any unknowns and make the simulation more accurate with minimal iterations. Assuming this AI is massively more intelligent than us, it isn’t unfeasible that it could find some zero-day exploits to take over all cloud computing to run the simulations, for example, so even the compute unavailability isn’t a great argument as to why it couldn’t do this.
Intelligence is a limited, partial substitute for knowledge. The problem with Aristotelian physics wasn't that Aristotle wasn't smart enough. It was that Aristotle lacked the ground truth knowledge to understand where he was wrong. An AI that can't run experiments will end up with elegant hallucinations of nanofactory blueprints but no actual nanofactory. (That's even assuming that Drexlerian nanotech is actually something that can be built. It's still an open question.)
All the computing power in the world is insufficient to simulate a living bacterium at the molecular level. We also don't have a molecular resolution map of bacteria in the first place. The AI will have to figure out how to make one. The AI can try to develop faster high fidelity approximations for molecular simulation, but that too requires calibration against experiments as well as a lot of simulation resources. If it seizes control of public cloud computing for its simulations people will notice and can just unplug cables. If its intrusions are subtle, only using little unnoticeable compute slices when machines are largely idle, then its simulations are going to get less compute resources in aggregate than researchers have now.
You're right about laser weapons vs laser research. A real problem is that non-lethal AI is very lucrative.
I haven't read the book, so I'm not familiar with which scenarios in particular Bostrom talks about. But I believe I'm versed enough in the issues with AI alignment to carry on an intelligent conversation about it. When you say that none of the scenarios weren't plausible enough, could you give an example as to what you mean?
To give an example from my end, I'm really just imagining a scenario where we give the AGI some sort of goal, such as curing cancer. We've thought about the ways things could go wrong, and used reinforcement learning from human feedback to make an AI that seems to have learned not to do anything that could cause serious harm to a human in training. But it turns out that it actually learned to game our rewards, and just learned not to do anything to harm humans that could be traced back to it. Similarly, we tried to train it not to acquire more resources and only use what we give it, but it just learned to hide its activity. We know reward hacking is extremely common in AIs trained this way. GPT-4 is still willing to provide users with forbidden information -- it hasn't actually internalized the goal of "don't provide people with information that they could likely use to harm themselves or others", it's just some superficially similar goal that we don't really understand.
So anyway, the AI didn't actually learn to care about humanity, only about putting on that appearance, like a sociopath that learned to fake emotions to blend in. It has learned the ability to reason and plan, in order to design virtual experiments to calculate protein folding, etc. We provide it with a plain language description of the problem we want solved, specifying a ton of caveats like:
* Don't take actions that would result in the greater chance of death of people with cancer. * Don't try to detect people with a high likelihood of cancer and kill people before they get it. * Try to come up with a solution to this that most people would be happy with.
We can't specify things like "don't do anything that might result in someone dying" because in previous tests the AI would refuse to try to cure cancer, in case someone who was cured then went on to kill someone else.
No matter how much we specify, it turns out that a sufficiently intelligent, creative AI can come up with a maximal solution. Say, it secretly creates a drug that it distributes through the water supply that makes most people happy with just about anything, then builds a bunch of suspended animation chambers where it stores people who have cancer for ever -- they can be revived, so they're not actually dead. It needs to keep things running forever, so of course it has to take over things in order to build enough of these chambers for everyone who ever lived. Now, that's not really possible, if people keep breeding and reproducing, so it ends up just sticking everyone into these chambers, which they're fine with because of the happy-drug. The earth continues silently through space, with all of humanity protected in suspended animation by our over-zealous AI. Oh, and any life that we mentioned in restrictions is also in suspended animation, while every other bit of life has been exterminated as the AI tried to maximize its resources in order to keep this state of affairs in place forever.
It's a thing where no matter how many restrictions you try to stick on to the problem, a sufficiently smart AI can probably come up with a better solution in one of the loopholes than it can by just doing the boring, human thing of trying to figure out an effective cancer drug that works in 98% of cases. The AI tries to get to 100%, which necessitates a more drastic solution. If you tell it just to come up with a drug with a greater than 90% effectiveness rate, it STILL wants to take over the world in order to acquire enough computing power to guarantee that the effectiveness is no less than 90%. We don't know how to build "satisficers" that are happy with good enough, and even if we could, "satisficers" will tend to build maximizers.
That's exactly what humans (which are kinda satisficers, sometimes) are doing by building AI -- we're trying to find maximal solutions to our problems, except I'm worried that we're not quite smart enough so we're gonna screw it up real bad.
I think we have a chance of building an AI that's aligned close enough not to kill us, as long as it's not that much smarter than us. But the smarter it gets, the more that 2-degree of misalignment matters. And they're already terrifyingly smart.
The AI doomism will help no one. Bing search is a good example. Very good simulation of an average human. Sometime same as hysterical as humans online... The thing is it is a simulation. The AI however smart it will be, will miss something very important for someone to want to eradicate humanity. It will miss the "will" and the erratic harmonal storms of our body and brain. It will miss the drive to want something. On the other hand, a good simulation of an average or even psychotic human on an AI capable to make this simulation smarter then all other humans, can be dangerous to humanity. Thus: STOP FEEDING AI with our biases and behavioral data, stop feeding it with the chat data and the literature data and the tones of online trash which is posted every day all around the internet. Stop feeding it with the data which will make it possible to simulate an average human. 32K context and possibility to focus on 3-4 key-points of the conversation now, will soon (like 2 years from now) change and will turn into 64k and 7-8 key-points. This is still not enough to build a long term strategy to eradicate humanity.
AI wouldn't need that to win at scale, I don't think it would play Drunken Kung Fu Master against humanity but rather perform millions of time efficient, calculated movements a'la StarCraft. Otherwise I agree these are traits that are beneficial for us to fight back. Don't mind me, just speculating.
Humanity got on the top of the food chain, not only because it is smarter and can turn predators into prey. Humans have a will to do so. Even animals have a will and biological needs. AI can't have it unless it simulates it. That's about it.