I think the idea is that it doesn't matter who builds the AGI first if we all end up dying anyway.
And the first step to prevent that from happening is for us to stop, and then hope that the other powerful countries will see that and stop as well. And everyone who's left can be forced to stop by the powerful countries who agreed to the treaty.
Yeah, I get that, but I what if Russia and China think "Well, we're fucked anyway, our population is aging faster than any other country in history, our economies are fucked by multiple factors including COVID, War, sanctions and technology bans, our countries won't make it past 100 more yeah, maybe not even 50 more years... if we achieve AGI before anyone else, or no one else does, there's maybe a 5% chance we can control it, and maybe a 60% or 70% chance it destroys all humans on earth or all life on earth... if we're lucky and we hit that 5% we'll rule the rest of the world, otherwise we were doomed as a country anyway..."
Then they end up racing to build an AGI we're not prepared for, and we all die.
The faint hope is that people are not crazy or suicidal enough to build a world-ending technology, that humanity will manage to see ourselves as being in the same boat, where we all live or all die as one. And just-not-do the thing that causes our extinction.
It's not a very realistic proposition, because even smart people often fail to understand these arguments (or just disagree with them), let alone the politicians (who are incentivized to ignore them), or the rest of the people. There's a lot at stake for the "winner" of the AGI race, who could build an AGI and align it successfully (which they can't do). And many countries are ruled by people who are clearly willing to do crazy and suicidal things for power.
That's why the author of this post doesn't put a lot of hope into the idea of his daughter growing up.
Yep, it's always a possibility, but it's better to reduce the chance any way we can. If the US banning it reduces the risk from 100% to 50%, that's fantastic.
I also don't think it's that likely that a leader would take those odds, if they understand the risk fully. The worry is that they won't understand or believe the risk, so that's where we should attack the problem, in my opinion--by putting out as much good information on AI risk, and combating misinformation, as best we can.
Comments
I think the idea is that it doesn't matter who builds the AGI first if we all end up dying anyway.
And the first step to prevent that from happening is for us to stop, and then hope that the other powerful countries will see that and stop as well. And everyone who's left can be forced to stop by the powerful countries who agreed to the treaty.
Yeah, I get that, but I what if Russia and China think "Well, we're fucked anyway, our population is aging faster than any other country in history, our economies are fucked by multiple factors including COVID, War, sanctions and technology bans, our countries won't make it past 100 more yeah, maybe not even 50 more years... if we achieve AGI before anyone else, or no one else does, there's maybe a 5% chance we can control it, and maybe a 60% or 70% chance it destroys all humans on earth or all life on earth... if we're lucky and we hit that 5% we'll rule the rest of the world, otherwise we were doomed as a country anyway..."
Then they end up racing to build an AGI we're not prepared for, and we all die.
The faint hope is that people are not crazy or suicidal enough to build a world-ending technology, that humanity will manage to see ourselves as being in the same boat, where we all live or all die as one. And just-not-do the thing that causes our extinction.
It's not a very realistic proposition, because even smart people often fail to understand these arguments (or just disagree with them), let alone the politicians (who are incentivized to ignore them), or the rest of the people. There's a lot at stake for the "winner" of the AGI race, who could build an AGI and align it successfully (which they can't do). And many countries are ruled by people who are clearly willing to do crazy and suicidal things for power.
That's why the author of this post doesn't put a lot of hope into the idea of his daughter growing up.
Yep, it's always a possibility, but it's better to reduce the chance any way we can. If the US banning it reduces the risk from 100% to 50%, that's fantastic.
I also don't think it's that likely that a leader would take those odds, if they understand the risk fully. The worry is that they won't understand or believe the risk, so that's where we should attack the problem, in my opinion--by putting out as much good information on AI risk, and combating misinformation, as best we can.