I can think of two ways. One, work under someone with that experience, solving those sorts of problems. The other, which is far more thrilling/terrifying, is to run into scaling problems, and have to figure out how to fix them. There's no teacher like experience.
As others have mentioned, though, the first step is to monitor and log everything. You want metrics on what people are doing, what your server is doing, what your network stack is doing, and everything in between. The more data you have, the more tools you have to a) locate, and b) fix problems. You don't know if something has improved if you don't have a baseline to measure it against!
Some of the tools I use include Mixpanel/Google Analytics (for understanding user behavior), Newrelic (for server-side runtime metrics), MongoDB profiling and the MySQL slow query logs (for database sticky spots), and Munin (for server/network monitoring). You'll want to build your own toolset depending on the needs of your network, but once you've figured out how to collect the data, you're a lot further than you think.
Comments
I can think of two ways. One, work under someone with that experience, solving those sorts of problems. The other, which is far more thrilling/terrifying, is to run into scaling problems, and have to figure out how to fix them. There's no teacher like experience.
As others have mentioned, though, the first step is to monitor and log everything. You want metrics on what people are doing, what your server is doing, what your network stack is doing, and everything in between. The more data you have, the more tools you have to a) locate, and b) fix problems. You don't know if something has improved if you don't have a baseline to measure it against!
Some of the tools I use include Mixpanel/Google Analytics (for understanding user behavior), Newrelic (for server-side runtime metrics), MongoDB profiling and the MySQL slow query logs (for database sticky spots), and Munin (for server/network monitoring). You'll want to build your own toolset depending on the needs of your network, but once you've figured out how to collect the data, you're a lot further than you think.