The more groups have AGI, the higher the chance that one of them figures out recursive self-improvement, and if that happens it's game over for everybody. Iterated prisoner's dilemma only works when all the prisoners have roughly the same power.
Why is it game over though? AI is already used for self-improvement, and will be used more as it becomes more capable, but that won't magically solve all the problems in the world and create omnipotent AI. If there are many people and many AI's they collectively will be stronger than the one most advanced, and cooperation will remain advantageous.
Comments
The more groups have AGI, the higher the chance that one of them figures out recursive self-improvement, and if that happens it's game over for everybody. Iterated prisoner's dilemma only works when all the prisoners have roughly the same power.
Why is it game over though? AI is already used for self-improvement, and will be used more as it becomes more capable, but that won't magically solve all the problems in the world and create omnipotent AI. If there are many people and many AI's they collectively will be stronger than the one most advanced, and cooperation will remain advantageous.