One thing I disagree with is that intelligence is just a very strong optimizer. It is that, but I don't think it's just that. I think it is very much a multi-modal / multi-paradigm thing
I agree with that. I didn't want to imply that you get intelligence by throwing enough compute at a gradient descent. It's multi-domain, cross-paradigm kind of optimization. But I find it worthwhile to deanthropomorphize intelligence, so that it's easier to appreciate how different can a powerful mind be from the ones we have.
and I think there's a ton of stuff we don't understand about the operation of our own minds. It's yet another reason I don't think human-level or beyond AI is imminent. We barely understand how we think.
I also don't think AI is imminent - hell, we can't even get a decent webapp generator working, we have a lot of research in front of us to build self-improving systems. But I share the concern of the FAI crowd that we may eventually get there, and when we do, it's important to do it right the first time - otherwise, a runaway optimization process may not give us another chance. As you said, we barely understand how we ourselves think - so it's good to figure that one out before someone manages to build an actual AI.
One thing I've learned by being a "smart person" is that intelligence is not everything. I do not believe that high intelligence or even super-intelligence would automatically yield power, wealth, influence, or anything else that we might fear, especially if it has a high chance of coming with various forms of baggage and trade-offs.
We can't assume that the apparent correlation between intelligence and mental illness would hold in non-humans -- or that it wouldn't. That's because we don't understand why that correlation exists. But there is one trade-off that I think is likely to be universal: the smarter you are, the more effort you seem to have to put into "meta" thinking like philosophy to keep yourself on track.
Think of it this way: it's easy to drive a Honda Civic, but a supercar can actually require performance driving classes to learn to drive it safely. Otherwise you can do things like accelerate into the car in front of you very easily, lose control, etc. because it does not drive like a commodity car.
Many people with very high IQs use their high intelligence to create elaborate delusions and rationalizations that land them in a ditch. That's a "meta" problem, a philosophical problem, not a problem with engine size, but it's one that probably gets worse as the motor gets bigger.
It's another factor that I think might place rate limits on the rate at which an AI could self-improve. Technically it's just a special case of the combinatorial search problem associated with improving intelligence beyond known local maxima.
An AI just thinking "hey I can just double my processing power and storage and get twice as smart!" might very easily end in some novel form of madness, not super-intelligence.
Comments
I agree with that. I didn't want to imply that you get intelligence by throwing enough compute at a gradient descent. It's multi-domain, cross-paradigm kind of optimization. But I find it worthwhile to deanthropomorphize intelligence, so that it's easier to appreciate how different can a powerful mind be from the ones we have.
I also don't think AI is imminent - hell, we can't even get a decent webapp generator working, we have a lot of research in front of us to build self-improving systems. But I share the concern of the FAI crowd that we may eventually get there, and when we do, it's important to do it right the first time - otherwise, a runaway optimization process may not give us another chance. As you said, we barely understand how we ourselves think - so it's good to figure that one out before someone manages to build an actual AI.
One thing I've learned by being a "smart person" is that intelligence is not everything. I do not believe that high intelligence or even super-intelligence would automatically yield power, wealth, influence, or anything else that we might fear, especially if it has a high chance of coming with various forms of baggage and trade-offs.
We can't assume that the apparent correlation between intelligence and mental illness would hold in non-humans -- or that it wouldn't. That's because we don't understand why that correlation exists. But there is one trade-off that I think is likely to be universal: the smarter you are, the more effort you seem to have to put into "meta" thinking like philosophy to keep yourself on track.
Think of it this way: it's easy to drive a Honda Civic, but a supercar can actually require performance driving classes to learn to drive it safely. Otherwise you can do things like accelerate into the car in front of you very easily, lose control, etc. because it does not drive like a commodity car.
Many people with very high IQs use their high intelligence to create elaborate delusions and rationalizations that land them in a ditch. That's a "meta" problem, a philosophical problem, not a problem with engine size, but it's one that probably gets worse as the motor gets bigger.
It's another factor that I think might place rate limits on the rate at which an AI could self-improve. Technically it's just a special case of the combinatorial search problem associated with improving intelligence beyond known local maxima.
An AI just thinking "hey I can just double my processing power and storage and get twice as smart!" might very easily end in some novel form of madness, not super-intelligence.