I once mishandled the api keys and endpoints for dev and production environments, effectively send dev alerts as prod and prod as dev.
Anyway The business was seasonal and entering spring (our big sales time), and I see a flood of errors coming. In on DEV environment.
I silenced them because I figured someone was working on it. Nope. Our customers were working that checkout process and the payment gateway was failing hard.
Yeah, so 12 hours later I get a fire drill call from VP finance asking why his morning reports report no sales.
I was too green to realize my mistake right away and took another 2 hours of bumping my head on things before seeing the glaring mistake.
Impact: 25 to 30k but maybe less as customer service made lots of follow up calls to close sales.
Love your customer service people. They are the ones who humanize your mistakes.
Comments
I once mishandled the api keys and endpoints for dev and production environments, effectively send dev alerts as prod and prod as dev.
Anyway The business was seasonal and entering spring (our big sales time), and I see a flood of errors coming. In on DEV environment.
I silenced them because I figured someone was working on it. Nope. Our customers were working that checkout process and the payment gateway was failing hard.
Yeah, so 12 hours later I get a fire drill call from VP finance asking why his morning reports report no sales.
I was too green to realize my mistake right away and took another 2 hours of bumping my head on things before seeing the glaring mistake.
Impact: 25 to 30k but maybe less as customer service made lots of follow up calls to close sales.
Love your customer service people. They are the ones who humanize your mistakes.