I might be missing something significant but it looks to me like the site was just static HTML and JavaScript with the JavaScript being updated at some rate.
If that's true why wouldn't you just do /xxxxx/clock.js and /xxxxx/data.js on S3 + CloudFront? The /version/ means caching is never going to get in your way because you're incrementing it each time you update.
Then the entire diagram is reduced to the web server + S3/CloudFront with software pushing new versions of the JavaScript.
Also is that for real about GAE cutting you off under load regardless of being paid? What's the point of using it to scale if there's a very low ceiling on how high you can go.
Having built the system I can probably answer some of those questions.
The cloudfront solution won't work because you need to push from s3 and as we found out during the previous sale that can take up to 20 minutes. That's no fun for anyone, believe me. I'd be pretty reluctant to ever use cloudfront again.
Gae cut you off at 500 rps - they downgrade you back off billing. You can apply to have the limit lifted, which we did but they didn't lift it for us.
There are other things too, like it can be a struggle to update content in a bucket when it's under significant load.
I'm sure everyone looks at the architecture and thinks it's overengineered but we got burned previously and just wanted as much stability and redundancy as possible. It's an interesting problem too, how to survive DDOSing yourself on a monumental scale.
Ultimately we ended up with a stable system at negligible cost and didn't get slated on twitter. That's good enough for me :)
Comments
I might be missing something significant but it looks to me like the site was just static HTML and JavaScript with the JavaScript being updated at some rate.
If that's true why wouldn't you just do /xxxxx/clock.js and /xxxxx/data.js on S3 + CloudFront? The /version/ means caching is never going to get in your way because you're incrementing it each time you update.
Then the entire diagram is reduced to the web server + S3/CloudFront with software pushing new versions of the JavaScript.
Also is that for real about GAE cutting you off under load regardless of being paid? What's the point of using it to scale if there's a very low ceiling on how high you can go.
Having built the system I can probably answer some of those questions.
The cloudfront solution won't work because you need to push from s3 and as we found out during the previous sale that can take up to 20 minutes. That's no fun for anyone, believe me. I'd be pretty reluctant to ever use cloudfront again.
Gae cut you off at 500 rps - they downgrade you back off billing. You can apply to have the limit lifted, which we did but they didn't lift it for us.
There are other things too, like it can be a struggle to update content in a bucket when it's under significant load.
I'm sure everyone looks at the architecture and thinks it's overengineered but we got burned previously and just wanted as much stability and redundancy as possible. It's an interesting problem too, how to survive DDOSing yourself on a monumental scale.
Ultimately we ended up with a stable system at negligible cost and didn't get slated on twitter. That's good enough for me :)