Not everybody works at eBay, GMail or Facebook scale.
Most applications fit very well in a single server. For example, Stack Overflow runs on a single instance of SQL Server, replicated to a slave in another data center. In such a case, the convenience of joins and transactions is priceless.
And even at scale, it makes sense to rely on joins and transactions. The perfect example is AdWords that runs of F1 and Spanner:
"Our users needed complex queries and joins, which meant they had to carefully shard their data, and resharding data without breaking applications was challenging."
Exactly. But many of us do have apps that are beyond the capabilities of a single instance.
And in this situation the rule of thumb is to do joins in your application layer so you can store different types data (e.g. graph/document/relational/unstructured) in different systems and easily cache where needed.
The fact that so many new databases have been created in the last decade suggests that there are a lot of people who do fall into this camp.
Yes. It's not an especially exotic level of performance. Especially if you're talking about applications that handle traffic from other machines, as opposed to humans. As websites go, Stack Overflow is of reasonable size, but as systems go it's small.
They have about 7.5m visits per day to Stack Overflow. That's about 86 requests per second, which perhaps at peak is several multiples larger than the average. I wouldn't consider that a particularly gargantuan website. That sounds like traffic that you could service with a reasonable fleet of web servers and database fleet, given caching. Quite a lot of actions on the site can be applied with eventual consistency, I'd imagine.
I don't mean to say that the system or the problem that it's solving is trivial - I am sure it is difficult to get right. As websites go, it might be large, but as systems go it's not particularly high traffic among systems that receive traffic from machines rather than humans. Imagine that you operate a data center, and you want to sample CPU, memory, etc. from your machines every minute. If you collect 50 samples per machine per minute, and you have 258 machines, you'll be handling 86 samples per second. Storing 86 samples per second into a time series database is probably considerably easier than SO's website rendering, but it goes to show that high-traffic or high-frequency systems are common in companies beyond small to medium size. It is easy for cross-cutting concerns like this to have massive request volumes, far greater than the human-generated traffic to any website.
7.5m visits per day. What does that mean, page loads? Stack Overflow isn't a static website. One page load is a lot of requests to the service. Stack Overflow is a very dynamic site, and a lot of requests are made after you actually load the page. I'm not sure you're accurately characterizing the kind of load that Stack Overflow is subjected to.
To anyone else reading: no, your use case probably isn't so special that the solution Stack Overflow arrived at just doesn't work for you.
There are also organizational requirements that push people away from joins (different teams responsible for warehousing different data in a large organization) and high performance applications which pre-summarize data (adtech).
Comments
How exactly do you think eBay, GMail, Facebook etc work ? They aren't relying on relational database joins.
If you want to write a truly scalable application you structure everything such that you do joins in your application layer.
http://highscalability.com/ebay-architecture
And in the case of MongoDB you avoid joins since it is a document database. You embed data instead.
Not everybody works at eBay, GMail or Facebook scale.
Most applications fit very well in a single server. For example, Stack Overflow runs on a single instance of SQL Server, replicated to a slave in another data center. In such a case, the convenience of joins and transactions is priceless.
And even at scale, it makes sense to rely on joins and transactions. The perfect example is AdWords that runs of F1 and Spanner:
"Our users needed complex queries and joins, which meant they had to carefully shard their data, and resharding data without breaking applications was challenging."
http://static.googleusercontent.com/media/research.google.co...
Exactly. But many of us do have apps that are beyond the capabilities of a single instance.
And in this situation the rule of thumb is to do joins in your application layer so you can store different types data (e.g. graph/document/relational/unstructured) in different systems and easily cache where needed.
The fact that so many new databases have been created in the last decade suggests that there are a lot of people who do fall into this camp.
A lot of us have apps that are beyond the performance requirements of Stack Overflow?
Yes. It's not an especially exotic level of performance. Especially if you're talking about applications that handle traffic from other machines, as opposed to humans. As websites go, Stack Overflow is of reasonable size, but as systems go it's small.
They have about 7.5m visits per day to Stack Overflow. That's about 86 requests per second, which perhaps at peak is several multiples larger than the average. I wouldn't consider that a particularly gargantuan website. That sounds like traffic that you could service with a reasonable fleet of web servers and database fleet, given caching. Quite a lot of actions on the site can be applied with eventual consistency, I'd imagine.
http://stackexchange.com/sites?view=list#traffic
I don't mean to say that the system or the problem that it's solving is trivial - I am sure it is difficult to get right. As websites go, it might be large, but as systems go it's not particularly high traffic among systems that receive traffic from machines rather than humans. Imagine that you operate a data center, and you want to sample CPU, memory, etc. from your machines every minute. If you collect 50 samples per machine per minute, and you have 258 machines, you'll be handling 86 samples per second. Storing 86 samples per second into a time series database is probably considerably easier than SO's website rendering, but it goes to show that high-traffic or high-frequency systems are common in companies beyond small to medium size. It is easy for cross-cutting concerns like this to have massive request volumes, far greater than the human-generated traffic to any website.
7.5m visits per day. What does that mean, page loads? Stack Overflow isn't a static website. One page load is a lot of requests to the service. Stack Overflow is a very dynamic site, and a lot of requests are made after you actually load the page. I'm not sure you're accurately characterizing the kind of load that Stack Overflow is subjected to.
To anyone else reading: no, your use case probably isn't so special that the solution Stack Overflow arrived at just doesn't work for you.
There are also organizational requirements that push people away from joins (different teams responsible for warehousing different data in a large organization) and high performance applications which pre-summarize data (adtech).