If you said that about a topic you're familiar with, it sounds silly: it seems that the vast vast majority of possible computer programs would have goals directly harmful to human goals (even modest ones).
The vast majority of possible medical treatments, proteins, DNAs, etc. would be directly harmful to human goals, and yet we keep on trying, cognizant and wary of these ramifications.
Granted, there are "irresponsible" researchers who don't understand that an AI might not have "gratitude" that prevents it from rm -rfing humanity like a bad shell script, but others work on 'Friendly AI' to mitigate or eliminate this concern.
A recursively self-improving intelligence seems far more likely to be an existential threat than virtually any other human creation. Even gray goo can be fought fairly easily with heat (yes, "fight it with fire" will be the slogan of the First Goo War... ;)), but if a hard takeoff is possible, the first AI will have nothing to oppose whatever its goals happen to be. The touted solution to this is Friendliness, as you note, but Friendliness appears to be a problem in the same class of difficulty (or more so) than AI itself, even if workable at all, which is uncertain. In my opinion, people who choose to work on Friendliness as a prerequisite to AI are simply guaranteeing that AI will be developed by someone else first.
the things you mentioned would be harmless, not harmful. at worst they would waste some resources. they certainly wouldn't requisition resources by force (unless you want to talk about ideology as a "technology" :)
Er, prions, viruses, bacteria, runaway breakout of monoculture -- these things use up resources, and have the potential to cause great harm. It's just that few of them are likely to rise to the level of existential threat, while in the field of recursively self-improving AI, the likelihood is strongly the other way.
Comments
If you said that about a topic you're familiar with, it sounds silly: it seems that the vast vast majority of possible computer programs would have goals directly harmful to human goals (even modest ones). The vast majority of possible medical treatments, proteins, DNAs, etc. would be directly harmful to human goals, and yet we keep on trying, cognizant and wary of these ramifications.
Granted, there are "irresponsible" researchers who don't understand that an AI might not have "gratitude" that prevents it from rm -rfing humanity like a bad shell script, but others work on 'Friendly AI' to mitigate or eliminate this concern.
http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...
A recursively self-improving intelligence seems far more likely to be an existential threat than virtually any other human creation. Even gray goo can be fought fairly easily with heat (yes, "fight it with fire" will be the slogan of the First Goo War... ;)), but if a hard takeoff is possible, the first AI will have nothing to oppose whatever its goals happen to be. The touted solution to this is Friendliness, as you note, but Friendliness appears to be a problem in the same class of difficulty (or more so) than AI itself, even if workable at all, which is uncertain. In my opinion, people who choose to work on Friendliness as a prerequisite to AI are simply guaranteeing that AI will be developed by someone else first.
Which is why this is important:
http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...
the things you mentioned would be harmless, not harmful. at worst they would waste some resources. they certainly wouldn't requisition resources by force (unless you want to talk about ideology as a "technology" :)
Er, prions, viruses, bacteria, runaway breakout of monoculture -- these things use up resources, and have the potential to cause great harm. It's just that few of them are likely to rise to the level of existential threat, while in the field of recursively self-improving AI, the likelihood is strongly the other way.
http://en.wikipedia.org/wiki/Morris_worm
http://online.sfsu.edu/~rone/GEessays/killervirus.html