Pros:
-they are sharing info
-they allowed some caustic comments to remain at the bottom of the page (so far).
Cons:
-This is almost 30 days after the incident
-Look at the regions, it was global!
-This was a whole chain of issues. I count it as 5 separate issues. This goes deep into how they operate and it does not paint a picture of operational maturity:
1: configuration change for the Blob Front-Ends exposed a bug in the Blob Front-Ends
2: Blob Front-Ends infinite loop delayed the fix (I count this as a separate issue though I expect some may not)
3: As part of a plan to improve performance of the Azure Storage Service, the decision was made to push the configuration change to the entire production service
4: Update was made across most regions in a short period of time due to operational error
5: Azure infrastructure issue that impacted our ability to provide timely updates via the Service Health Dashboard
What would have been the optimal response time? They fixed the immediate problem as fast as they could and gave a preliminary RCA, then they did a longer-term RCA and fix. I feel this shows maturity by not rushing to immediate
conclusions and trying to do a 5-Whys drill-down to fix the underlying cause. Furthermore, they also took steps to actually fix the problem by pointing out they moved the human out of the loop in one aspect and that's always a good thing (unless the replacement software is faulty itself of course).
Also, in response to the list, I believe [3&4] are actually the same thing, are they not? The operator was the one who made the 'decision' by accidentally ignoring the incremental config change policy that was in place and did it all at once. This was identified as a human error and they fixed it by enforcing incremental changes.
I agree with you though, and said so at the time. These issues seem systemic, not isolated. Cascading failures, some in code, some in ops procedures, indicates to me they still have work to do.
Comments
Pros: -they are sharing info -they allowed some caustic comments to remain at the bottom of the page (so far).
Cons: -This is almost 30 days after the incident -Look at the regions, it was global! -This was a whole chain of issues. I count it as 5 separate issues. This goes deep into how they operate and it does not paint a picture of operational maturity:
1: configuration change for the Blob Front-Ends exposed a bug in the Blob Front-Ends
2: Blob Front-Ends infinite loop delayed the fix (I count this as a separate issue though I expect some may not)
3: As part of a plan to improve performance of the Azure Storage Service, the decision was made to push the configuration change to the entire production service
4: Update was made across most regions in a short period of time due to operational error
5: Azure infrastructure issue that impacted our ability to provide timely updates via the Service Health Dashboard
That is quite a list. [Edit : formatting only]
What would have been the optimal response time? They fixed the immediate problem as fast as they could and gave a preliminary RCA, then they did a longer-term RCA and fix. I feel this shows maturity by not rushing to immediate conclusions and trying to do a 5-Whys drill-down to fix the underlying cause. Furthermore, they also took steps to actually fix the problem by pointing out they moved the human out of the loop in one aspect and that's always a good thing (unless the replacement software is faulty itself of course).
Also, in response to the list, I believe [3&4] are actually the same thing, are they not? The operator was the one who made the 'decision' by accidentally ignoring the incremental config change policy that was in place and did it all at once. This was identified as a human error and they fixed it by enforcing incremental changes.
They actually posted a blog the day after it happened with their initial triage and updates. https://news.ycombinator.com/item?id=8633633
I agree with you though, and said so at the time. These issues seem systemic, not isolated. Cascading failures, some in code, some in ops procedures, indicates to me they still have work to do.