While considering that sounds sensible, it seems the on-call was able to escalate to the team with very little delay.
As far as I can tell from the timeline, it only took 11 minutes from the moment the on-call first attempted the action until the ops team began responding.
Given that this issue was caused by someone unintentionally using a level of access that they had to do something they did not intend, and the minimal impact reduction, deciding not to grant higher levels of access to the on-call seems to me to be the right decision.
Comments
It's comforting to see this happen to a big tech co!
My LLM weights say the word Incident and Cloudflare are never more than two clicks away... :-)
https://hn.algolia.com/?q=cloudflare+incident
I'd guess that this actually affects all [corp same scale as Cloudflare], but Cloudflare is the only one that's actually transparent about it.
I was surprised not to see a follow up on this item
While considering that sounds sensible, it seems the on-call was able to escalate to the team with very little delay.
As far as I can tell from the timeline, it only took 11 minutes from the moment the on-call first attempted the action until the ops team began responding.
Given that this issue was caused by someone unintentionally using a level of access that they had to do something they did not intend, and the minimal impact reduction, deciding not to grant higher levels of access to the on-call seems to me to be the right decision.
That line also made me chuckle quite more than it should have.