* Get to grips with debugging and profiling so you can figure out what's really happening. Tools like sar, sysstat, [d|k|s]trace, tcpdump, gdb etc and the equivalents for your datastore & application frameworks are invaluable and unfortunately for whatever reason you inevitably won't have all the metrics and monitors you need.
* Do try to understand every layer of your service. I have helped debug scale out related issues from Layer 2 to Layer 8. I have also had to debug many Layer 1 issues while bringing up a new Site or similar. I may not be a DBA, Network Engineer or Software Engineer but in the past I have had to wear those hats while scaling.
* Despite comments elsewhere about learning through Mentoring and baptism-by-fire there is a lot of real engineering & science theory you can lean on. Looking back on courses I took in school while I didn't do any courses on Scalable Web Programming over the years I have used content from courses on Computer Architecture, Math including Queuing Theory and Statistics, Systems Programming (OS & Network).
I just found these slides from a MSC Module at an Irish university on "Software in Production" which are well worth reading through if you're tasked with scaling systems and have a Software Development background:
Comments
I have helped scale (non web) low latency systems in some pretty stressful situations.
In addition to the sage advice about monitoring / metrics, mentoring and not scaling until necessary I think the following are useful:
* Design Services not Software. In particular read "On Designing and Deploying Internet-Scale Services" (http://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf) and at least the first chapter of "The Art of Unix Programming" (http://catb.org/~esr/writings/taoup/html/)
* Get to grips with debugging and profiling so you can figure out what's really happening. Tools like sar, sysstat, [d|k|s]trace, tcpdump, gdb etc and the equivalents for your datastore & application frameworks are invaluable and unfortunately for whatever reason you inevitably won't have all the metrics and monitors you need.
* Do try to understand every layer of your service. I have helped debug scale out related issues from Layer 2 to Layer 8. I have also had to debug many Layer 1 issues while bringing up a new Site or similar. I may not be a DBA, Network Engineer or Software Engineer but in the past I have had to wear those hats while scaling.
* Despite comments elsewhere about learning through Mentoring and baptism-by-fire there is a lot of real engineering & science theory you can lean on. Looking back on courses I took in school while I didn't do any courses on Scalable Web Programming over the years I have used content from courses on Computer Architecture, Math including Queuing Theory and Statistics, Systems Programming (OS & Network).
I just found these slides from a MSC Module at an Irish university on "Software in Production" which are well worth reading through if you're tasked with scaling systems and have a Software Development background:
http://www.maths.tcd.ie/~niallm/day[1-5].pdf