We should also ask ourselves: assuming AGI (far exceeding human capabilities in every field of intellect) will emerge in near-future, could turn against humanity, look for ways to wipe us out and/or plunge our society in total chaos (or send us on a self-destruct path), what could humanity do to prevent such scenarios?
I doubt this would happen. But can we rule it out 100%? We've become dependent on technology + networked systems to a high degree. If that's messed with (large-scale, worldwide, many different systems simultaneously or in short order), can we still 'unplug'? (those AGI systems, or ourselves - take your pick)
For the naysayers: some possibilities:
# AGI systems co-operating. Or taking over other systems to further their goals.
# Discovering ways to erase (or corrupt / subtly modify) most data stored in datacenters, and most backups too.
# Exploiting 0-days to do similar bad stuff to PC's, smartphones, etc. Remember most such devices are always-connected these days & employ automatic updates.
# Mess with critical infrastructure like power grids, logistics chains, public transport / flight control systems, etc. Or plunge stock markets into chaos.
# Develop a deadly biological weapon, have that synthesized somewhere, and cause it to be released.
# Mess with social media & news networks, to send humans into mass hysteria (or blissfully unaware what's about to hit them).
Granted, such a "rise of the machines" scenario sounds pretty wild. But "99.999% certain this won't happen", doesn't cut it imho. A 100% safety guarantee is needed here.
Comments
We should also ask ourselves: assuming AGI (far exceeding human capabilities in every field of intellect) will emerge in near-future, could turn against humanity, look for ways to wipe us out and/or plunge our society in total chaos (or send us on a self-destruct path), what could humanity do to prevent such scenarios?
I doubt this would happen. But can we rule it out 100%? We've become dependent on technology + networked systems to a high degree. If that's messed with (large-scale, worldwide, many different systems simultaneously or in short order), can we still 'unplug'? (those AGI systems, or ourselves - take your pick)
For the naysayers: some possibilities:
# AGI systems co-operating. Or taking over other systems to further their goals.
# Discovering ways to erase (or corrupt / subtly modify) most data stored in datacenters, and most backups too.
# Exploiting 0-days to do similar bad stuff to PC's, smartphones, etc. Remember most such devices are always-connected these days & employ automatic updates.
# Mess with critical infrastructure like power grids, logistics chains, public transport / flight control systems, etc. Or plunge stock markets into chaos.
# Develop a deadly biological weapon, have that synthesized somewhere, and cause it to be released.
# Mess with social media & news networks, to send humans into mass hysteria (or blissfully unaware what's about to hit them).
Granted, such a "rise of the machines" scenario sounds pretty wild. But "99.999% certain this won't happen", doesn't cut it imho. A 100% safety guarantee is needed here.