While considering that sounds sensible, it seems the on-call was able to escalate to the team with very little delay.
As far as I can tell from the timeline, it only took 11 minutes from the moment the on-call first attempted the action until the ops team began responding.
Given that this issue was caused by someone unintentionally using a level of access that they had to do something they did not intend, and the minimal impact reduction, deciding not to grant higher levels of access to the on-call seems to me to be the right decision.
Cloudflare has nice services that they make available to a lot of people for free.
At the same time, this reminds me the cloud is someone else's computer, and seeking input and ideas of how to have a failover with other services or something.
Does anyone know of a setup or design that can shim in a bit of redundancy with something like this? Using one cloud does kind of tie you to them a bit more.
Not really -- the circular dependency issue happened 28 minutes after the customer impact started. That issue was a cause of the outage being double the length it could have been, but not an original cause of the outage.
Comments
It's comforting to see this happen to a big tech co!
My LLM weights say the word Incident and Cloudflare are never more than two clicks away... :-)
https://hn.algolia.com/?q=cloudflare+incident
I'd guess that this actually affects all [corp same scale as Cloudflare], but Cloudflare is the only one that's actually transparent about it.
I was surprised not to see a follow up on this item
While considering that sounds sensible, it seems the on-call was able to escalate to the team with very little delay.
As far as I can tell from the timeline, it only took 11 minutes from the moment the on-call first attempted the action until the ops team began responding.
Given that this issue was caused by someone unintentionally using a level of access that they had to do something they did not intend, and the minimal impact reduction, deciding not to grant higher levels of access to the on-call seems to me to be the right decision.
That line also made me chuckle quite more than it should have.
You can tell a company really builds using their own products when an abuse system can take them offline!
Cloudflare has nice services that they make available to a lot of people for free.
At the same time, this reminds me the cloud is someone else's computer, and seeking input and ideas of how to have a failover with other services or something.
Does anyone know of a setup or design that can shim in a bit of redundancy with something like this? Using one cloud does kind of tie you to them a bit more.
Surely it was the new intern.
Who just got a million dollars worth of training.
The training was done by AI. In fact, the intern is also AI.
appears to be caused by service circular dependency
Not really -- the circular dependency issue happened 28 minutes after the customer impact started. That issue was a cause of the outage being double the length it could have been, but not an original cause of the outage.