A good summary with one exception: the monitoring, instrumentation and logging didn't get enough attention. The failure is the norm so firsts and foremost you want to know when a failure occurs and then you need to be able to investigate what went wrong. You should literally monitor/instrument everything: every API call, every DB access, every page rendering should include code to monitor latency, result codes, payload size, etc. Every error (even benign) should be logged preferably with stack traces. Any unexpected condition should be checked and logged. All the instrumentation data should be graphed and stored for a long period of time so you can analyze the impact of your code changes on system performance and correlate it with system failures.
Comments
A good summary with one exception: the monitoring, instrumentation and logging didn't get enough attention. The failure is the norm so firsts and foremost you want to know when a failure occurs and then you need to be able to investigate what went wrong. You should literally monitor/instrument everything: every API call, every DB access, every page rendering should include code to monitor latency, result codes, payload size, etc. Every error (even benign) should be logged preferably with stack traces. Any unexpected condition should be checked and logged. All the instrumentation data should be graphed and stored for a long period of time so you can analyze the impact of your code changes on system performance and correlate it with system failures.