The US military is playing with fire creating AI that will carry out the mission at all costs. This scenario is literally the plot of EVERY sci-fi warning about misused AI.
These aren't operational systems, these are research systems. No need for the hyperbole. The reason for these tests is to figure out exactly these kinds of failure modes (many of which have already been predicted) and how to handle them.
The crux of the issue is "how can we be certain that we've addressed all the failure modes?" This isn't like proving the correctness of a program where we can rely on exhaustivness or universal quantification.
"It worked in the simulation," is the embodied AI equivalent of "It worked on my machine." It's not acceptable.
As a first step, if addressing the issues were the intent rather than stoking AI fears, they would mention details about the AI/LLM and promoting being used
How confident are you that the issues can be addressed? How confident would you need to be to endorse deploying it out in the wild? How confident would you need to be that no "unforeseen" issues would arise?
I believe it's necessary to establish these levels of confidence even before discussing specific failure cases.
We would normally discuss failures in terms of base-rate comparison, but we don't have one for "how often autonomous weapons are unaligned?" Enumerating the failure modes isn't a substitute for the empirical distribution of failure mode probabilities.
Comments
These aren't operational systems, these are research systems. No need for the hyperbole. The reason for these tests is to figure out exactly these kinds of failure modes (many of which have already been predicted) and how to handle them.
The crux of the issue is "how can we be certain that we've addressed all the failure modes?" This isn't like proving the correctness of a program where we can rely on exhaustivness or universal quantification.
"It worked in the simulation," is the embodied AI equivalent of "It worked on my machine." It's not acceptable.
As a first step, if addressing the issues were the intent rather than stoking AI fears, they would mention details about the AI/LLM and promoting being used
How confident are you that the issues can be addressed? How confident would you need to be to endorse deploying it out in the wild? How confident would you need to be that no "unforeseen" issues would arise?
I believe it's necessary to establish these levels of confidence even before discussing specific failure cases.
We would normally discuss failures in terms of base-rate comparison, but we don't have one for "how often autonomous weapons are unaligned?" Enumerating the failure modes isn't a substitute for the empirical distribution of failure mode probabilities.