When you need to scale, you'll find you'll be able to.
The question is probably more, "What can you do today to make scaling less painful when I need to, but without wasting effort today?"
The answer to that is to monitor everything.
Scaling is about being able to spot the bottle neck in the system and to remove it or work around it.
Unfortunately that's a hard thing to do, you need to be aware of everything: networking layers, hardware, software, how things work (or don't work) together. You've got to be able to holistically see the entire application and understand how one part impacts the rest.
Only obsessive monitoring can help you there unless you're a little bit savant and can visualise the whole system.
When you identify a bottle neck, you re-design and remove it.
How to make that easier? I find thinking of large systems in terms of a network of systems that each do "one thing well" helps. It makes identifying bottle necks easier as you cannot have doubt where a bottle neck is if a machine (or it's network) is running hot and it really does just a single task.
So the only advice I give people in advance of them having pain points is to not mix the use of a part of the system. If you have a database server, make sure that is the only thing that server does.
For anyone trying to learn how to scale, the way to scale differs from system to system. Very few people could tell you anything, though a lot of highscalability.com is general enough for web apps that you can build up an arsenal of approaches to common problems and turn to those when you spot bottle necks emerge.
Ultimately: you can't learn how to scale until you have to, but you can design your system such that it all does one thing well and is obsessively monitored... at least then when you need to you're in a good place to be able to react to that demand.
Don't just monitor general things like CPU, I/O... monitor what your application does as well. The point of monitoring is to give you the ability to spot the interconnectedness of symptoms to their cause.
Knowing that CPU spiked shortly after networking became saturated isn't as helpful as knowing that your calls were doing requests of a large number of records and returning more data than needed. So if you fail to monitor your actual application too, you only have half of the picture.
Comments
When you need to scale, you'll find you'll be able to.
The question is probably more, "What can you do today to make scaling less painful when I need to, but without wasting effort today?"
The answer to that is to monitor everything.
Scaling is about being able to spot the bottle neck in the system and to remove it or work around it.
Unfortunately that's a hard thing to do, you need to be aware of everything: networking layers, hardware, software, how things work (or don't work) together. You've got to be able to holistically see the entire application and understand how one part impacts the rest.
Only obsessive monitoring can help you there unless you're a little bit savant and can visualise the whole system.
When you identify a bottle neck, you re-design and remove it.
How to make that easier? I find thinking of large systems in terms of a network of systems that each do "one thing well" helps. It makes identifying bottle necks easier as you cannot have doubt where a bottle neck is if a machine (or it's network) is running hot and it really does just a single task.
So the only advice I give people in advance of them having pain points is to not mix the use of a part of the system. If you have a database server, make sure that is the only thing that server does.
For anyone trying to learn how to scale, the way to scale differs from system to system. Very few people could tell you anything, though a lot of highscalability.com is general enough for web apps that you can build up an arsenal of approaches to common problems and turn to those when you spot bottle necks emerge.
Ultimately: you can't learn how to scale until you have to, but you can design your system such that it all does one thing well and is obsessively monitored... at least then when you need to you're in a good place to be able to react to that demand.
I failed to give you a starting point on monitoring, try these:
http://graphite.wikidot.com/ (I prefer this)
http://munin-monitoring.org/ (widely used and fairly easy to find resources for)
Don't just monitor general things like CPU, I/O... monitor what your application does as well. The point of monitoring is to give you the ability to spot the interconnectedness of symptoms to their cause.
Knowing that CPU spiked shortly after networking became saturated isn't as helpful as knowing that your calls were doing requests of a large number of records and returning more data than needed. So if you fail to monitor your actual application too, you only have half of the picture.