Skip to content

Comment on Final Root Cause Analysis of Nov 18 Azure Service Interruptionparent

Comments

"This is tantamount to saying "yeah sometimes our airplanes crash, so from now on we'll just make sure we have less passengers ride in the newer models"."

Things break. You can not design something to be perfect. That is why there is redundancy in every critical system. You are better off having your system gracefully recover from failure, than trying to design the system perfectly. That might mean spreading the traffic across multiple instances within one data center so when one breaks (for any reason), the others pick up the slack. The next level is spreading across multiple data centers so if one place goes down, you have another pick up the slack. Arguably the next level would be going across multiple providers, but to me that seems like overkill.

Given that, if Microsoft rolls it out to 5% of servers and those crap out, that is roughly equivalent for individual customers that are properly spread over multiple instances as a spate of harddrive failures. This only breaks down when they roll the broken stuff out to 100% at once.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.