On that subject, have you seen aristocrats at work? That isn't even a line of defence, there is a 2-3 generation timer (optimistically) before we're looking at well educated halfwits. If they've got sentient robot commanders and there is any scope for them to go rogue, we don't have a human institution that'll stop that and certainly not something with a small group of - emphasis on the quote marks - "elites".
If we're talking existential risk, this is like Rome v. Carthage. Humans aren't going to come back once we're a bit behind. Although I don't think there is much to do and humans are so dodgy that it isn't even obvious that we should resist if the opportunity comes to replace us with something that can be consistently rational and intelligent.
Because AI exists in the universe with outside forces and has the same risks as anything else for dissolution of its complex state ("life," if we are being generous): run out of energy (starve), extinction (Earth blows up) or dissassembly (i.e. a virus).
The biggest weakness of AI as a self-sustaining life form is the lack of resilience to something like a virus. Having something live forever is easy - reproduction and evolution are an answer to provide diversity and robustness to viruses. Like if COVID was 100% deadly it wouldn't have wiped out humanity because of natural immunity in some % of the population.
So if AI is to persist it would at least need to co-opt evolution as a survival strategy, or come up with something better.
Comments
On that subject, have you seen aristocrats at work? That isn't even a line of defence, there is a 2-3 generation timer (optimistically) before we're looking at well educated halfwits. If they've got sentient robot commanders and there is any scope for them to go rogue, we don't have a human institution that'll stop that and certainly not something with a small group of - emphasis on the quote marks - "elites".
If we're talking existential risk, this is like Rome v. Carthage. Humans aren't going to come back once we're a bit behind. Although I don't think there is much to do and humans are so dodgy that it isn't even obvious that we should resist if the opportunity comes to replace us with something that can be consistently rational and intelligent.
Humans are about as rational and intelligent as you'd expect from any evolved species.
Which is to say, our intelligence is mostly tuned to increase the number of descendants we have.
If AIs are subject to evolutionary forces, the same will be true for them.
Why do you think AI will be subject to the same evolutionary forces as organic life?
Because AI exists in the universe with outside forces and has the same risks as anything else for dissolution of its complex state ("life," if we are being generous): run out of energy (starve), extinction (Earth blows up) or dissassembly (i.e. a virus).
The biggest weakness of AI as a self-sustaining life form is the lack of resilience to something like a virus. Having something live forever is easy - reproduction and evolution are an answer to provide diversity and robustness to viruses. Like if COVID was 100% deadly it wouldn't have wiped out humanity because of natural immunity in some % of the population.
So if AI is to persist it would at least need to co-opt evolution as a survival strategy, or come up with something better.