I am confused at how Waymo engineering can be so robust as to yield an astonishingly good safety record, and yet so slapdash as to happily drive into deep water.
I feel this is actually somewhat straightforward. I assume deep water on roadways is not commonly in the training set, because frankly it isn't common in real life, and when it is common people do not drive and do not gather that training data. As a result the proper response has not adequately been beaten into the models. There are probably also challenges of world-sensing, since water can act as a mirror, and maybe other complications. So waymos are bad at handling deep water on roadways. However, deep water on roadways is also not common in the areas where waymos are deployed. As a result, waymo's have a great safety record, and at the same time they make mistakes that are obvious to a human.
A common criticism of AI discourse is that people act as if LLM's "think". I don't want to be a vocabulary purist, but I suspect that's related to the astonishment here -- the Waymo doesn't know what flooding is, it doesn't fear drowning, it doesn't think. So unless it's been repeatedly trained, or a special case has been hard coded by manual effort, it doesn't know that flooded roadways are dangerous.
I have made a lot of assumptions here, and I don't truthfully know what the training data looks like. Feel free to push back if you think my assumptions are wrong. I'd especially be interested if somebody can show that water on roadways _is_ in the training data
People drive into deep water all the time - some states specifically have laws making them financially liable for the cost of rescue because it’s such a stupid thing to do. But still, they do it.
I think the fact that there are laws about it is not good evidence that it should be in the training data. Laws often cover weird edge cases, and if an edge case happens 50 times in 100 years, there's likely a law covering it. At the same time, that's probably not enough occurrences for it to naturally end up in a dataset -- the edge case would probably need to be intentionally sought out. I'm not saying that driving-in-deep water only happens 50 times in 100 years, it's certainly more common than that, I'm just saying that despite laws on the topic, it may still be too rare to be well represented in training data. For example, in real life I've only seen a car drive into deep water once. Even if we include recordings that I've seen, that would maybe bring it up to 20?
Without a good way of getting it into the dataset via simulation, or a more speculative approach (world models?), edge cases and unusual circumstances could cause failures.
I think my broader point is people don't do so well in unusual circumstances either: blizzards, heavy rain, dust storms, etc. They'll hydroplane, drive into stopped traffic, etc. We need to decide if we'll hold self-driving cars to some unreasonable standard of perfection or accept them once they are X safer than a human benchmark, even if they still have Y rate of failure per million miles.
My thinking here is that Waymo has logged hundreds of millions of miles at this point (and even more in sim), and there are a lot of nines in their safety record. So even the rare edge cases should have come up. You make a good point that the deep water scenario may be not only rare, but also under-represented in the training data. On the other hand, you'd think they would have thought to explicitly train for it; it's not that obscure of a possibility.
Ya, this does strike me as pretty high on the list of the thing I'd intentionally seek out if I was doing this job, along with object-in-the-road, icy road, tornado/hurricane, wildfire, and perhaps hail.
Comments
I feel this is actually somewhat straightforward. I assume deep water on roadways is not commonly in the training set, because frankly it isn't common in real life, and when it is common people do not drive and do not gather that training data. As a result the proper response has not adequately been beaten into the models. There are probably also challenges of world-sensing, since water can act as a mirror, and maybe other complications. So waymos are bad at handling deep water on roadways. However, deep water on roadways is also not common in the areas where waymos are deployed. As a result, waymo's have a great safety record, and at the same time they make mistakes that are obvious to a human.
A common criticism of AI discourse is that people act as if LLM's "think". I don't want to be a vocabulary purist, but I suspect that's related to the astonishment here -- the Waymo doesn't know what flooding is, it doesn't fear drowning, it doesn't think. So unless it's been repeatedly trained, or a special case has been hard coded by manual effort, it doesn't know that flooded roadways are dangerous.
I have made a lot of assumptions here, and I don't truthfully know what the training data looks like. Feel free to push back if you think my assumptions are wrong. I'd especially be interested if somebody can show that water on roadways _is_ in the training data
People drive into deep water all the time - some states specifically have laws making them financially liable for the cost of rescue because it’s such a stupid thing to do. But still, they do it.
I think the fact that there are laws about it is not good evidence that it should be in the training data. Laws often cover weird edge cases, and if an edge case happens 50 times in 100 years, there's likely a law covering it. At the same time, that's probably not enough occurrences for it to naturally end up in a dataset -- the edge case would probably need to be intentionally sought out. I'm not saying that driving-in-deep water only happens 50 times in 100 years, it's certainly more common than that, I'm just saying that despite laws on the topic, it may still be too rare to be well represented in training data. For example, in real life I've only seen a car drive into deep water once. Even if we include recordings that I've seen, that would maybe bring it up to 20?
Oh yeah, I’m not saying it should be in the training data - driving into the water would be a bad data collection strategy.
My point was more that a self-driving car going into water isn’t some unreasonably bad action, as people do it all the time.
If we set the bar for self-driving to be as safe as a human driver, or even 2x as safe, this behavior would still happen.
Ahhhh, that makes sense, that is an interesting point — I think you’re saying it’s just not high risk enough for them to make sure it’s got coverage
Without a good way of getting it into the dataset via simulation, or a more speculative approach (world models?), edge cases and unusual circumstances could cause failures.
I think my broader point is people don't do so well in unusual circumstances either: blizzards, heavy rain, dust storms, etc. They'll hydroplane, drive into stopped traffic, etc. We need to decide if we'll hold self-driving cars to some unreasonable standard of perfection or accept them once they are X safer than a human benchmark, even if they still have Y rate of failure per million miles.
(I'm the author of the blog post)
My thinking here is that Waymo has logged hundreds of millions of miles at this point (and even more in sim), and there are a lot of nines in their safety record. So even the rare edge cases should have come up. You make a good point that the deep water scenario may be not only rare, but also under-represented in the training data. On the other hand, you'd think they would have thought to explicitly train for it; it's not that obscure of a possibility.
Ya, this does strike me as pretty high on the list of the thing I'd intentionally seek out if I was doing this job, along with object-in-the-road, icy road, tornado/hurricane, wildfire, and perhaps hail.