It's nice to see detail like this on high-traffic, high-risk environments.
I'm curious about their provisions for cross-datacenter failover. The article mentions haproxy being ready to direct requests to a different datacenter as well as ELB spanning availability zones. I'd expect a failover option entirely outside AWS as well, with short-TTL DNS ready to make the switch.
I'm also not sure what value varnish brings to the table when the Apaches are just serving a small number of flat files. It seems like unnecessary complexity -- by the same logic found earlier in the article -- when a well-tuned webserver will serve flat files at a comparable rate. Maybe the Apache configuration for this workload was sufficiently different to make it an unwanted risk.
If you look at the current webserving stacks propagating out there, wev'e gone from apache to multi-layered stuff, haproxy/varnish/nginx/apache/mongrel/rails/ various fcgi/etc..... and the thing to keep in mind is there is overlap between most of these projects.
Apache can, one way or the other, do most of what the others can do - possibly not as well, and at the risk of a much more complex configuration.
Haproxy does one thing and does it really,really well - it's great for dealing with load balancing, concurrency limiting per user defined resource, and identifying and routing incoming requests to the right infrastructure. It's really good at this - that's what it does.
Varnish does one thing and does it really, really well - it caches content and serves it up (usually out of memory, but even if it's swapped, it's optimized) to keep the load off your application servers. It has various optimizations built into make it really good at this.
So - in this case, the answer might be "not much - we could just put the apache's out front behind haproxy" - but it appears putting the varnish server out front with a 5 second cache dropped the load on the application servers (in this case apache serving static files that it receives over rsync - dont' forget the rsync part - resources are needed for that). This might result in smoother output for the end user, rather than something hitting a node that's busy servicing an rsync update.
It may also be their engineers are very familiar with the haproxy/varnish front end setup, as it presumably exists in their current day to day operation as well.. so the people responsible for keeping things up probably decided "Yes, we'd like to keep it there - it makes our lives easier."
There is an operational anti-pattern in there - removing too many elements from a known system is also a kind of added cmoplexity - all your troubleshooting methods disappear.
Their goal here was to de-couple the dynamic elements from the event-driven side of things and turn them into something more resilient (and less flexible) for a short time to deal with unknown and unpredicably large load.
It does. Apache is strictly one process per connection, varnish handles hundreds, thousands or even tens of thousands of connections with just one process so the overhead is minimal.
Apache has not strictly been one process per connection since 2.0. See the worker MPM. I would have chosen nginx, but Apache can be configured as a capable static file server.
It is pure optimization - they said it gives them a performance boost and smooths things out.
In their "flat-file" setup though - they are deliberately configured so that, should varnish fail (which was one of their concerns) - they don't actually need it - they could just have haproxy immediateley start hitting the apache servers directly.
So I know nothing about haproxy, but what sort of system load does it generate?
They say they ran it on a EC2 micro instance, and the micro instances are specifically designed to handle spikes, not continuous heavy load. In fact, they intentionally throttle under continuous heavy load.
I guess it must have been the right choice for them, I was just surprised to see it was a micro. Would that typically be the right way to go?
HAProxy is very light. We put it between nginx and our webservers for better failure handling, load-balancing, and some additional logging detail. On a small system (single AMD 1226) it's averaged 3% CPU over the last 15 days while handling ~300M requests.
The micro instance was fronting the render farm. As far as I can tell the article doesn't specify the instance types in use for any of the other components.
Comments
It's nice to see detail like this on high-traffic, high-risk environments.
I'm curious about their provisions for cross-datacenter failover. The article mentions haproxy being ready to direct requests to a different datacenter as well as ELB spanning availability zones. I'd expect a failover option entirely outside AWS as well, with short-TTL DNS ready to make the switch.
I'm also not sure what value varnish brings to the table when the Apaches are just serving a small number of flat files. It seems like unnecessary complexity -- by the same logic found earlier in the article -- when a well-tuned webserver will serve flat files at a comparable rate. Maybe the Apache configuration for this workload was sufficiently different to make it an unwanted risk.
If you look at the current webserving stacks propagating out there, wev'e gone from apache to multi-layered stuff, haproxy/varnish/nginx/apache/mongrel/rails/ various fcgi/etc..... and the thing to keep in mind is there is overlap between most of these projects.
Apache can, one way or the other, do most of what the others can do - possibly not as well, and at the risk of a much more complex configuration.
Haproxy does one thing and does it really,really well - it's great for dealing with load balancing, concurrency limiting per user defined resource, and identifying and routing incoming requests to the right infrastructure. It's really good at this - that's what it does.
Varnish does one thing and does it really, really well - it caches content and serves it up (usually out of memory, but even if it's swapped, it's optimized) to keep the load off your application servers. It has various optimizations built into make it really good at this.
So - in this case, the answer might be "not much - we could just put the apache's out front behind haproxy" - but it appears putting the varnish server out front with a 5 second cache dropped the load on the application servers (in this case apache serving static files that it receives over rsync - dont' forget the rsync part - resources are needed for that). This might result in smoother output for the end user, rather than something hitting a node that's busy servicing an rsync update. It may also be their engineers are very familiar with the haproxy/varnish front end setup, as it presumably exists in their current day to day operation as well.. so the people responsible for keeping things up probably decided "Yes, we'd like to keep it there - it makes our lives easier."
There is an operational anti-pattern in there - removing too many elements from a known system is also a kind of added cmoplexity - all your troubleshooting methods disappear.
Their goal here was to de-couple the dynamic elements from the event-driven side of things and turn them into something more resilient (and less flexible) for a short time to deal with unknown and unpredicably large load.
Doesn't Varnish handle many times more concurrent clients than apache does, at significantly lower system load? It could just be pure optimization.
It does. Apache is strictly one process per connection, varnish handles hundreds, thousands or even tens of thousands of connections with just one process so the overhead is minimal.
Apache has not strictly been one process per connection since 2.0. See the worker MPM. I would have chosen nginx, but Apache can be configured as a capable static file server.
Even then each thread is still one connection (according to the apache docs).
Except in the Event MPM
It is pure optimization - they said it gives them a performance boost and smooths things out.
In their "flat-file" setup though - they are deliberately configured so that, should varnish fail (which was one of their concerns) - they don't actually need it - they could just have haproxy immediateley start hitting the apache servers directly.
So I know nothing about haproxy, but what sort of system load does it generate?
They say they ran it on a EC2 micro instance, and the micro instances are specifically designed to handle spikes, not continuous heavy load. In fact, they intentionally throttle under continuous heavy load.
I guess it must have been the right choice for them, I was just surprised to see it was a micro. Would that typically be the right way to go?
HAProxy is very light. We put it between nginx and our webservers for better failure handling, load-balancing, and some additional logging detail. On a small system (single AMD 1226) it's averaged 3% CPU over the last 15 days while handling ~300M requests.
The micro instance was fronting the render farm. As far as I can tell the article doesn't specify the instance types in use for any of the other components.
Ah! I see that now, thanks for pointing that out!