I don't really necessarily agree with that. I think it's important to think about your application at scale and if it makes sense for you to start with a slightly more complex beginning, but a much easier time when you hit scale. A lot of my job is helping people who are stuck on a single box with an increasingly complex rapidly scaling application and no real well thought out plan for how to start splitting that app into clustered components.
I think it's important to think about your application at scale and if it makes sense for you to start with a slightly more complex beginning, but a much easier time when you hit scale.
That's a text-book example of premature optimization.
A lot of my job is helping people who are stuck on a single box with an increasingly complex rapidly scaling application and no real well thought out plan for how to start splitting that app into clustered components.
That's a fine time to start thinking about how to solve that problem and there are plenty of good, battle tested solutions out there. The first one is cache as much as you can, that will buy you a lot of time to get to a more scale-friendly setup.
Remember that if they had spent their precious runway time on thinking about scaling instead of customer acquisition that they probably would not have a company at all at this stage, rather than a solvable scaling problem.
Are you comfortable sharing your application's actual availability? I'd love to throw away my preconceptions about redundancy but I find it hard to believe you can reach five 9s with a single machine.
The biggest issue in reliability are hard drives, power supplies, network interfaces and power infrastructure. At least, over the last 16 years of operating a series of websites those have been the main causes of trouble.
Uptime does not say much about service uptime, for instance, if the network uplink on one of those machines is down then the users will experience an outage, having a redundant, multi-data center setup would guard against such a situation.
But that would immediately introduce a whole pile of other problems. For instance, in a multi-master setup it would be quite difficult to recover if the only thing that went down was the peering link between the two data centers, with both locations still accessible from the public internet.
In that situation there is a 50/50 chance that my simple-but-dumb strategy would not even be noticed and a 50/50 chance that we'd be down.
That doens't mean there are no situations where such a distributed setup would be warranted but from where I'm sitting the economy just isn't there.
Having regular hardware is no reason by itself why such hardware could not be reliable, regular applications stacks perform remarkably well and the weak points in networking are just as weak when they are connecting otherwise reliable components across WAN links as when they are connecting outsiders to your co-location facility.
Once you start scaling up and/or out the whole equation changes and you need to invest a lot more into planning and testing your setup. Most people find out that their distributed setup was a little less distributed than they thought it was when the first outage hits them. This stuff is very hard to get right and most companies do not operate at a scale where this is a requirement, nor do they have a 100% uptime requirement. Of course we'd all like to pretend we're that important but that's a nonsense argument, the only way you're going to get to 100% is by spending an infinity of money. Everything can go down.
Sorry, counter experience here. I've seen too many shops with over exuberant usage expectations waist (and I mean waist) months and months of developer time when they really needed to be working on other features. Couple of times it killed the company because they spent so much time working on (non-existant) scalability issues they did get around to deploying or addressing the actual need of the end user. Too much emphasis was placed on getting everything perfect the first time around, and not staging that out over time as need arises.
Now, when dealing with what you are talking about, typically I've seen that from developers just making bad decisions in the development process full-stop. E.g. Moving the database to a different server should amount to a config change, not a code change.
Comments
I don't really necessarily agree with that. I think it's important to think about your application at scale and if it makes sense for you to start with a slightly more complex beginning, but a much easier time when you hit scale. A lot of my job is helping people who are stuck on a single box with an increasingly complex rapidly scaling application and no real well thought out plan for how to start splitting that app into clustered components.
That's a text-book example of premature optimization.
That's a fine time to start thinking about how to solve that problem and there are plenty of good, battle tested solutions out there. The first one is cache as much as you can, that will buy you a lot of time to get to a more scale-friendly setup.
Remember that if they had spent their precious runway time on thinking about scaling instead of customer acquisition that they probably would not have a company at all at this stage, rather than a solvable scaling problem.
Premature high availability does suck, you're right, my bad.
What a strange comment.
Complex does not equal high-availability, simple does not equate low availability.OS uptime does not equal application uptime. :)
Are you comfortable sharing your application's actual availability? I'd love to throw away my preconceptions about redundancy but I find it hard to believe you can reach five 9s with a single machine.
Most sites and most apps don't need five 9s, and I doubt that OP is saying that a single machine is going to give anyone that.
Must be amazing to have such perfect networking and applications. I've never worked on a project like that before.
You think that's rare?
That's really not all that rare.The biggest issue in reliability are hard drives, power supplies, network interfaces and power infrastructure. At least, over the last 16 years of operating a series of websites those have been the main causes of trouble.
Uptime does not say much about service uptime, for instance, if the network uplink on one of those machines is down then the users will experience an outage, having a redundant, multi-data center setup would guard against such a situation.
But that would immediately introduce a whole pile of other problems. For instance, in a multi-master setup it would be quite difficult to recover if the only thing that went down was the peering link between the two data centers, with both locations still accessible from the public internet.
In that situation there is a 50/50 chance that my simple-but-dumb strategy would not even be noticed and a 50/50 chance that we'd be down.
That doens't mean there are no situations where such a distributed setup would be warranted but from where I'm sitting the economy just isn't there.
Having regular hardware is no reason by itself why such hardware could not be reliable, regular applications stacks perform remarkably well and the weak points in networking are just as weak when they are connecting otherwise reliable components across WAN links as when they are connecting outsiders to your co-location facility.
Once you start scaling up and/or out the whole equation changes and you need to invest a lot more into planning and testing your setup. Most people find out that their distributed setup was a little less distributed than they thought it was when the first outage hits them. This stuff is very hard to get right and most companies do not operate at a scale where this is a requirement, nor do they have a 100% uptime requirement. Of course we'd all like to pretend we're that important but that's a nonsense argument, the only way you're going to get to 100% is by spending an infinity of money. Everything can go down.
Sorry, counter experience here. I've seen too many shops with over exuberant usage expectations waist (and I mean waist) months and months of developer time when they really needed to be working on other features. Couple of times it killed the company because they spent so much time working on (non-existant) scalability issues they did get around to deploying or addressing the actual need of the end user. Too much emphasis was placed on getting everything perfect the first time around, and not staging that out over time as need arises.
Now, when dealing with what you are talking about, typically I've seen that from developers just making bad decisions in the development process full-stop. E.g. Moving the database to a different server should amount to a config change, not a code change.
No, you probably mean “waste”.