Comment on Full technical details on Asana's worst outageparentComments−jwatte10yThe detail was right there: debugging something in security caused massive logging which caused CPU bottlenecking.Performance is the hardest thing to integration test for. Keeping careful track of CPU/memory/network/disk load with automated alerts can help.(Fancy systems like running a traffic replica can help, too, but at a much higher cost.)−marcog1OP10yWe actually have a traffic replica (dark client) setup for the new webserver architecture we are gradually migrating to. It likely would have caught this before deploying to users.
Comments
The detail was right there: debugging something in security caused massive logging which caused CPU bottlenecking.
Performance is the hardest thing to integration test for. Keeping careful track of CPU/memory/network/disk load with automated alerts can help.
(Fancy systems like running a traffic replica can help, too, but at a much higher cost.)
We actually have a traffic replica (dark client) setup for the new webserver architecture we are gradually migrating to. It likely would have caught this before deploying to users.