The biggest issue in reliability are hard drives, power supplies, network interfaces and power infrastructure. At least, over the last 16 years of operating a series of websites those have been the main causes of trouble.
Uptime does not say much about service uptime, for instance, if the network uplink on one of those machines is down then the users will experience an outage, having a redundant, multi-data center setup would guard against such a situation.
But that would immediately introduce a whole pile of other problems. For instance, in a multi-master setup it would be quite difficult to recover if the only thing that went down was the peering link between the two data centers, with both locations still accessible from the public internet.
In that situation there is a 50/50 chance that my simple-but-dumb strategy would not even be noticed and a 50/50 chance that we'd be down.
That doens't mean there are no situations where such a distributed setup would be warranted but from where I'm sitting the economy just isn't there.
Having regular hardware is no reason by itself why such hardware could not be reliable, regular applications stacks perform remarkably well and the weak points in networking are just as weak when they are connecting otherwise reliable components across WAN links as when they are connecting outsiders to your co-location facility.
Once you start scaling up and/or out the whole equation changes and you need to invest a lot more into planning and testing your setup. Most people find out that their distributed setup was a little less distributed than they thought it was when the first outage hits them. This stuff is very hard to get right and most companies do not operate at a scale where this is a requirement, nor do they have a 100% uptime requirement. Of course we'd all like to pretend we're that important but that's a nonsense argument, the only way you're going to get to 100% is by spending an infinity of money. Everything can go down.
Comments
Must be amazing to have such perfect networking and applications. I've never worked on a project like that before.
You think that's rare?
That's really not all that rare.The biggest issue in reliability are hard drives, power supplies, network interfaces and power infrastructure. At least, over the last 16 years of operating a series of websites those have been the main causes of trouble.
Uptime does not say much about service uptime, for instance, if the network uplink on one of those machines is down then the users will experience an outage, having a redundant, multi-data center setup would guard against such a situation.
But that would immediately introduce a whole pile of other problems. For instance, in a multi-master setup it would be quite difficult to recover if the only thing that went down was the peering link between the two data centers, with both locations still accessible from the public internet.
In that situation there is a 50/50 chance that my simple-but-dumb strategy would not even be noticed and a 50/50 chance that we'd be down.
That doens't mean there are no situations where such a distributed setup would be warranted but from where I'm sitting the economy just isn't there.
Having regular hardware is no reason by itself why such hardware could not be reliable, regular applications stacks perform remarkably well and the weak points in networking are just as weak when they are connecting otherwise reliable components across WAN links as when they are connecting outsiders to your co-location facility.
Once you start scaling up and/or out the whole equation changes and you need to invest a lot more into planning and testing your setup. Most people find out that their distributed setup was a little less distributed than they thought it was when the first outage hits them. This stuff is very hard to get right and most companies do not operate at a scale where this is a requirement, nor do they have a 100% uptime requirement. Of course we'd all like to pretend we're that important but that's a nonsense argument, the only way you're going to get to 100% is by spending an infinity of money. Everything can go down.