Keeping humans in the loop doesn't do as much as we think, because humans have a natural inclination to trust computers. In a high-stress situation, such as a battlefield, a human will defer to a computer even if the computer reports information that, under normal circumstances, would be obviously false. In 1988, for example, the USS Vincennes shot down a civilian airliner because its AI targeting system had misidentified it as an enemy aircraft. Giving individual humans access to an off-switch is a pretty suboptimal way to prevent AIpocalypse.
Comments
Keeping humans in the loop doesn't do as much as we think, because humans have a natural inclination to trust computers. In a high-stress situation, such as a battlefield, a human will defer to a computer even if the computer reports information that, under normal circumstances, would be obviously false. In 1988, for example, the USS Vincennes shot down a civilian airliner because its AI targeting system had misidentified it as an enemy aircraft. Giving individual humans access to an off-switch is a pretty suboptimal way to prevent AIpocalypse.