We do a write on every ad view, so there is a bit more complexity here. Most of those writes should be atomic (eg. each ad will only be viewed once) -- but there is a case where an ad might get viewed, then clicked, and we need the view data to know if the click is valid.
Just an aside, but if the primary goal was to increase capacity (transactions per second), actually performing a write on every ad view would be the first thing I'd seek to eliminate. Is it possible to keep the details as a class/object in RAM, update an in-memory array or cache, and do a bulk write to the DB once every, say, 5 minutes? Even if you lost 0.5% of your data (and I would expect your actual losses would be much lower), we're talking ad clicks, not bank transactions. Eliminating the DB write and confirmation on every request, especially over the network, could easily speed up responses and therefore capacity by 5x or more.
We may have to eventually do that. That also might make more sense for handling requests across continents. Every 5 minutes still seems like not frequent enough but presumably the same approach could be used to write every 5-10s.
What we have considered is not doing any synchronous writes and just queuing it up and handling it async. There's definitely some questions about whether our current approach will scale 10x but it should be fine for the next 2-3x.
Just to shed a bit more light, we break things up into figuring out which ad to show and then handling when that ad is actually seen. The second part is mostly async already but the first part is the harder part. You want to choose the best ad for the content, the geographic targeting needs to match, and the ad campaign has to have budget left on the hour/day/total. If we only checked the budget every 5 minutes, that would be a problem. At some point, multiple servers need to know that there's still budget on a campaign and this requires some amount of synchronization.
Sure, I don't know your exact business, those numbers were just guesses. Still, if you are doing half of your stated capacity (50 requests per second), then a single database write every 5 seconds reduces that traffic 249/250. Much of it depends how often the data needs to be read.
This is a common pattern, however. The easiest thing to set up is to do a customer database query for every request, while the far more efficient option, in many, many cases, is to cache the 1000 most-likely queries and serve up static or pre-rendered data. Not cache like one update per day, but perhaps one update per minute. Only you and your team can decide whether it would cost more to serve up a few free ads (over a customer's budget) because the cache was a minute or two old, vs doubling/tripling your infrastructure costs in order to make the system more precise. Sounds like you've got a good roadmap for making that decision.
I hate when HN'ers just chime in and say you're shit why didn't you do this.
So hopefully I'm not that totally rude person, no-shade intended pointing out that if for some random reason you haven't seen ClickHouse you should check it out.
Solves problems that I parsed from your blog post and originally built for that use case.
100 recs/s for an ad network is super duper mega tiny so I hope you're successful and get bigger!
Comments
Don't most major cloud providers offer multi-region replication to start solving the Europe/Asia latency issue?
Their data model seems very simple and appears to assume that there is a single consistent source of truth all the time.
We do a write on every ad view, so there is a bit more complexity here. Most of those writes should be atomic (eg. each ad will only be viewed once) -- but there is a case where an ad might get viewed, then clicked, and we need the view data to know if the click is valid.
Just an aside, but if the primary goal was to increase capacity (transactions per second), actually performing a write on every ad view would be the first thing I'd seek to eliminate. Is it possible to keep the details as a class/object in RAM, update an in-memory array or cache, and do a bulk write to the DB once every, say, 5 minutes? Even if you lost 0.5% of your data (and I would expect your actual losses would be much lower), we're talking ad clicks, not bank transactions. Eliminating the DB write and confirmation on every request, especially over the network, could easily speed up responses and therefore capacity by 5x or more.
We may have to eventually do that. That also might make more sense for handling requests across continents. Every 5 minutes still seems like not frequent enough but presumably the same approach could be used to write every 5-10s.
What we have considered is not doing any synchronous writes and just queuing it up and handling it async. There's definitely some questions about whether our current approach will scale 10x but it should be fine for the next 2-3x.
Just to shed a bit more light, we break things up into figuring out which ad to show and then handling when that ad is actually seen. The second part is mostly async already but the first part is the harder part. You want to choose the best ad for the content, the geographic targeting needs to match, and the ad campaign has to have budget left on the hour/day/total. If we only checked the budget every 5 minutes, that would be a problem. At some point, multiple servers need to know that there's still budget on a campaign and this requires some amount of synchronization.
Sure, I don't know your exact business, those numbers were just guesses. Still, if you are doing half of your stated capacity (50 requests per second), then a single database write every 5 seconds reduces that traffic 249/250. Much of it depends how often the data needs to be read.
This is a common pattern, however. The easiest thing to set up is to do a customer database query for every request, while the far more efficient option, in many, many cases, is to cache the 1000 most-likely queries and serve up static or pre-rendered data. Not cache like one update per day, but perhaps one update per minute. Only you and your team can decide whether it would cost more to serve up a few free ads (over a customer's budget) because the cache was a minute or two old, vs doubling/tripling your infrastructure costs in order to make the system more precise. Sounds like you've got a good roadmap for making that decision.
I hate when HN'ers just chime in and say you're shit why didn't you do this.
So hopefully I'm not that totally rude person, no-shade intended pointing out that if for some random reason you haven't seen ClickHouse you should check it out.
Solves problems that I parsed from your blog post and originally built for that use case.
100 recs/s for an ad network is super duper mega tiny so I hope you're successful and get bigger!
I did check out ClickHouse but I haven't gotten a chance to load more real data to give it the full test. It's definitely on the todo.