I imagine this can be remedied by slow rolling non critical updates out so that the entire fleet doesn't get upended by Bobby Tables at once. You could trivially observe the daily change in accidents/collisions/whatever and adjust fire
Works great until the problem condition is not evenly distributed in place and time. Imagine that the release goes out in June but can’t handle icy roads; or the release goes out and can’t handle leap years; or it goes to cars in Iowa but has a problem interpreting ocean mist.
That remedy already applies to massive profitable services like Google and Facebook, yet they still have outages caused by sloppy configuration pushes.
Comments
I imagine this can be remedied by slow rolling non critical updates out so that the entire fleet doesn't get upended by Bobby Tables at once. You could trivially observe the daily change in accidents/collisions/whatever and adjust fire
Works great until the problem condition is not evenly distributed in place and time. Imagine that the release goes out in June but can’t handle icy roads; or the release goes out and can’t handle leap years; or it goes to cars in Iowa but has a problem interpreting ocean mist.
That remedy already applies to massive profitable services like Google and Facebook, yet they still have outages caused by sloppy configuration pushes.