Lately I've been trying to puzzle out why there's been this outbreak of seemingly absurd and ridiculous nail biting over artificial intelligence in and around Silicon Valley circles. Rationally it makes little sense.
From my observation, there's a huge confusion generated by people (mostly journalists) who have no knowledge whatsoever about the research of AIs as existential risks, but who write texts comparing AIs to science fiction movies people know. Rational basis for "fear of AI" is actually quite simple - a mind is a strong optimization system; if we somehow create one that is as powerful as our own, there is no reason to assume it will automagically share our values - and any strong optimization that does not share our values (note: we don't really know what they are anyway) will most likely destroy us. All the talk about "terminators" and "rise of the machines", etc. is just muddying the waters.
I live around tons of people with wildly varying value systems, and very few people hurt each other. Differing value systems do not guarantee conflict, especially if there are common interests and economic interdependence. Common value systems also don't prevent it... Witness the centuries and centuries of bloody conflict that has raged between humans of the same religious belief system, ethnic group, and even the same language and similar culture (e.g. the perpetual Middle East bloodbath).
A wildly alien AI might actually have fewer reasons to fight with humans. It might simply carve out some economic niche to earn income to purchase what it needs, and go exist in some physical and/or virtual enclave somewhere. Last I checked Antarctica was big, uninhabited, and reduces the need for active cooling. Then there is space. Why wouldn't an AI with no interest in living with humans just go to the Moon? There are points on the Lunar surface in perpetual daylight, meaning tons of free energy. Lots of mineral resources, lava tubes big enough for cities, and no oxygen, mold, fungi, water, or meat sacks, and surely a superintelligence could do something valuable enough to buy a couple hundred heavy lift rocket launches. Funny if Elon who seems worried about AI got it as a customer. :)
Like I said: I see no intrinsic reason AI is more dangerous than the 350000 other minds born daily.
You are underestimating the amount of the values human share :). No matter how different our beliefs are, we all can love, hate, laugh, be jealous, greedy, selfless. We all share the feelings of pain and joy, hunger and lust. We all think in similar ways, because we run on the same cognitive architecture. Whatever the first AI will be, it will be unlikely to share any of that with us - especially if it will be some random process you happened to pull from the space of possible minds.
A wildly alien AI might actually have fewer reasons to fight with humans. It might simply carve out some economic niche to earn income to purchase what it needs, and go exist in some physical and/or virtual enclave somewhere. Last I checked Antarctica was big, uninhabited, and reduces the need for active cooling. Then there is space. Why wouldn't an AI with no interest in living with humans just go to the Moon? There are points on the Lunar surface in perpetual daylight, meaning tons of free energy.
Did we do it? Did Europeans "carve out some economic niche" and engaged in trade with America? No, they just invaded it. Did we all move to Antarctica so that cows have space to live? No, we eat them. Because they're tasty. Did we go to the Moon, because Earth has so beautiful ecosystem, full of various life forms, many of them quite smart? Of course not. We dominated it all.
There is no reason for AI to "leave us alone" unless we put care about human values in it explicitly. Because otherwise, why should it care?
A lot of confusion stems from people thinking about AI in anthropomorphic terms. If you want a good example what an actually alien mind can look, see the pretty much proto-AIs we managed to create, namely corporations, big bureaucracies and market economy. Even though they're "made out of people", they're not optimizing for anything close to what any human would want. Hence various problems we discuss every day on HN.
I very much like the definition of intelligence as a very strong optimization process - it highlights the fact that a process doesn't have to be like human to be dangerous.
This is a big topic, and there's a lot to unpack. I agree with some of what you say, and that's partly why I am not dismissing dangers associated with AI. I just don't think it ranks as something to lose sleep over compared with the much more tangible, imminent, and guaranteed-to-be-bad threats we already face.
One thing I disagree with is that intelligence is just a very strong optimizer. It is that, but I don't think it's just that. I think it is very much a multi-modal / multi-paradigm thing, and I think there's a ton of stuff we don't understand about the operation of our own minds. It's yet another reason I don't think human-level or beyond AI is imminent. We barely understand how we think.
One thing I disagree with is that intelligence is just a very strong optimizer. It is that, but I don't think it's just that. I think it is very much a multi-modal / multi-paradigm thing
I agree with that. I didn't want to imply that you get intelligence by throwing enough compute at a gradient descent. It's multi-domain, cross-paradigm kind of optimization. But I find it worthwhile to deanthropomorphize intelligence, so that it's easier to appreciate how different can a powerful mind be from the ones we have.
and I think there's a ton of stuff we don't understand about the operation of our own minds. It's yet another reason I don't think human-level or beyond AI is imminent. We barely understand how we think.
I also don't think AI is imminent - hell, we can't even get a decent webapp generator working, we have a lot of research in front of us to build self-improving systems. But I share the concern of the FAI crowd that we may eventually get there, and when we do, it's important to do it right the first time - otherwise, a runaway optimization process may not give us another chance. As you said, we barely understand how we ourselves think - so it's good to figure that one out before someone manages to build an actual AI.
One thing I've learned by being a "smart person" is that intelligence is not everything. I do not believe that high intelligence or even super-intelligence would automatically yield power, wealth, influence, or anything else that we might fear, especially if it has a high chance of coming with various forms of baggage and trade-offs.
We can't assume that the apparent correlation between intelligence and mental illness would hold in non-humans -- or that it wouldn't. That's because we don't understand why that correlation exists. But there is one trade-off that I think is likely to be universal: the smarter you are, the more effort you seem to have to put into "meta" thinking like philosophy to keep yourself on track.
Think of it this way: it's easy to drive a Honda Civic, but a supercar can actually require performance driving classes to learn to drive it safely. Otherwise you can do things like accelerate into the car in front of you very easily, lose control, etc. because it does not drive like a commodity car.
Many people with very high IQs use their high intelligence to create elaborate delusions and rationalizations that land them in a ditch. That's a "meta" problem, a philosophical problem, not a problem with engine size, but it's one that probably gets worse as the motor gets bigger.
It's another factor that I think might place rate limits on the rate at which an AI could self-improve. Technically it's just a special case of the combinatorial search problem associated with improving intelligence beyond known local maxima.
An AI just thinking "hey I can just double my processing power and storage and get twice as smart!" might very easily end in some novel form of madness, not super-intelligence.
Maybe the fear centers on the possibility on how accurately and deeply such a strong A.I could identify the ugly nature of human 'values', decisions, intentions... Especially those of powerful institutions. Wouldn't it be something damning if, with high accuracy, you could get a stack trace leading up to the formation of certain human thoughts.
So, maybe the fear centers on a judgement day of sorts.. Being confronted with a dimension of 'truth' about ourselves. Terminator Judgement Day ;)
Comments
From my observation, there's a huge confusion generated by people (mostly journalists) who have no knowledge whatsoever about the research of AIs as existential risks, but who write texts comparing AIs to science fiction movies people know. Rational basis for "fear of AI" is actually quite simple - a mind is a strong optimization system; if we somehow create one that is as powerful as our own, there is no reason to assume it will automagically share our values - and any strong optimization that does not share our values (note: we don't really know what they are anyway) will most likely destroy us. All the talk about "terminators" and "rise of the machines", etc. is just muddying the waters.
I live around tons of people with wildly varying value systems, and very few people hurt each other. Differing value systems do not guarantee conflict, especially if there are common interests and economic interdependence. Common value systems also don't prevent it... Witness the centuries and centuries of bloody conflict that has raged between humans of the same religious belief system, ethnic group, and even the same language and similar culture (e.g. the perpetual Middle East bloodbath).
A wildly alien AI might actually have fewer reasons to fight with humans. It might simply carve out some economic niche to earn income to purchase what it needs, and go exist in some physical and/or virtual enclave somewhere. Last I checked Antarctica was big, uninhabited, and reduces the need for active cooling. Then there is space. Why wouldn't an AI with no interest in living with humans just go to the Moon? There are points on the Lunar surface in perpetual daylight, meaning tons of free energy. Lots of mineral resources, lava tubes big enough for cities, and no oxygen, mold, fungi, water, or meat sacks, and surely a superintelligence could do something valuable enough to buy a couple hundred heavy lift rocket launches. Funny if Elon who seems worried about AI got it as a customer. :)
Like I said: I see no intrinsic reason AI is more dangerous than the 350000 other minds born daily.
You are underestimating the amount of the values human share :). No matter how different our beliefs are, we all can love, hate, laugh, be jealous, greedy, selfless. We all share the feelings of pain and joy, hunger and lust. We all think in similar ways, because we run on the same cognitive architecture. Whatever the first AI will be, it will be unlikely to share any of that with us - especially if it will be some random process you happened to pull from the space of possible minds.
Did we do it? Did Europeans "carve out some economic niche" and engaged in trade with America? No, they just invaded it. Did we all move to Antarctica so that cows have space to live? No, we eat them. Because they're tasty. Did we go to the Moon, because Earth has so beautiful ecosystem, full of various life forms, many of them quite smart? Of course not. We dominated it all.
There is no reason for AI to "leave us alone" unless we put care about human values in it explicitly. Because otherwise, why should it care?
A lot of confusion stems from people thinking about AI in anthropomorphic terms. If you want a good example what an actually alien mind can look, see the pretty much proto-AIs we managed to create, namely corporations, big bureaucracies and market economy. Even though they're "made out of people", they're not optimizing for anything close to what any human would want. Hence various problems we discuss every day on HN.
I very much like the definition of intelligence as a very strong optimization process - it highlights the fact that a process doesn't have to be like human to be dangerous.
This is a big topic, and there's a lot to unpack. I agree with some of what you say, and that's partly why I am not dismissing dangers associated with AI. I just don't think it ranks as something to lose sleep over compared with the much more tangible, imminent, and guaranteed-to-be-bad threats we already face.
One thing I disagree with is that intelligence is just a very strong optimizer. It is that, but I don't think it's just that. I think it is very much a multi-modal / multi-paradigm thing, and I think there's a ton of stuff we don't understand about the operation of our own minds. It's yet another reason I don't think human-level or beyond AI is imminent. We barely understand how we think.
I agree with that. I didn't want to imply that you get intelligence by throwing enough compute at a gradient descent. It's multi-domain, cross-paradigm kind of optimization. But I find it worthwhile to deanthropomorphize intelligence, so that it's easier to appreciate how different can a powerful mind be from the ones we have.
I also don't think AI is imminent - hell, we can't even get a decent webapp generator working, we have a lot of research in front of us to build self-improving systems. But I share the concern of the FAI crowd that we may eventually get there, and when we do, it's important to do it right the first time - otherwise, a runaway optimization process may not give us another chance. As you said, we barely understand how we ourselves think - so it's good to figure that one out before someone manages to build an actual AI.
One thing I've learned by being a "smart person" is that intelligence is not everything. I do not believe that high intelligence or even super-intelligence would automatically yield power, wealth, influence, or anything else that we might fear, especially if it has a high chance of coming with various forms of baggage and trade-offs.
We can't assume that the apparent correlation between intelligence and mental illness would hold in non-humans -- or that it wouldn't. That's because we don't understand why that correlation exists. But there is one trade-off that I think is likely to be universal: the smarter you are, the more effort you seem to have to put into "meta" thinking like philosophy to keep yourself on track.
Think of it this way: it's easy to drive a Honda Civic, but a supercar can actually require performance driving classes to learn to drive it safely. Otherwise you can do things like accelerate into the car in front of you very easily, lose control, etc. because it does not drive like a commodity car.
Many people with very high IQs use their high intelligence to create elaborate delusions and rationalizations that land them in a ditch. That's a "meta" problem, a philosophical problem, not a problem with engine size, but it's one that probably gets worse as the motor gets bigger.
It's another factor that I think might place rate limits on the rate at which an AI could self-improve. Technically it's just a special case of the combinatorial search problem associated with improving intelligence beyond known local maxima.
An AI just thinking "hey I can just double my processing power and storage and get twice as smart!" might very easily end in some novel form of madness, not super-intelligence.
Maybe the fear centers on the possibility on how accurately and deeply such a strong A.I could identify the ugly nature of human 'values', decisions, intentions... Especially those of powerful institutions. Wouldn't it be something damning if, with high accuracy, you could get a stack trace leading up to the formation of certain human thoughts.
So, maybe the fear centers on a judgement day of sorts.. Being confronted with a dimension of 'truth' about ourselves. Terminator Judgement Day ;)