Running a master in us-west and having read replica in europe, what could go wrong ...
My advice is don't do those things in the blog and keep the DB and the app in the same region it will save you so many problems. If you can't just use a DB that was designed for that like Spanner or Cockroach.
I'm a big fan of "pinning" mechanisms where any time a user performs a write to your database you set a cookie that lasts for a few seconds and causes any subsequent requests from that user to go to the lesser, not a replica.
This solves the stale-read-after-write problem and ensures users will see any changes that they have personally made.
This does assume a mostly-reads, occasional-write application (which defines most of the applications I've worked on in my career).
So given that kind of application, what are the problems I'm not considering that I should look out for with a read-replica on a different continent?
Of the many things that can go wrong, the one I would be most worried about would be replication lag causing the disk on the primary to fill with WAL logs bringing down the primary, or if you have limited the WAL logs, the replica not being able to keep up and detaching from the primary causing stale reads.
Also long distances will make replication latency increase so unless _all_ requests are sent to the primary region you will likely have a read after write state data situation unless you go out of your way to ensure consistency
Stale replicas are a "stop serving requests from a particular region" problem. We pretty aggressively monitor these and consider things broken if replication is lagged more than about 10s.
One thing about running thousands of DB clusters (which we did at Compose.com before this) is, these problems exist even without geo replication. And they exist at such a scale that you have to be good at handling them. Geo replicas are _more_ prone to these kinds of issues but we already had to be good at handling them.
That is good and should prevent against bringing down the primary. There is still the issue with being able to get that region back online once replication is severed, which isn't always possible even with co-located systems.
Is the WAL log situation you describe here a likely function of both distance (hence latency) and volume of writes? So if I have a high volume of writes AND I'm trying to replicate them half way around the globe I'm more at risk of running out of WAL log space on the leader?
Yeah, if you are doing global replication you are much more likely to fall behind so you should have aggressive monitoring on the replica to remove it if replica lag gets to high, that will prevent the primary from retaining WAL files for too long.
It also means you will need to be able to resync the replica from a backup once write volume calms down.
All the things that can go wrong with globally distributed databases. You can just run an HA cluster in a single region if you're looking to keep it simple, just the same way you can run a single-region app on Fly if you like.
We try to go out of our way to say that this stuff isn't a perfect fit for every application or a "best" way to do things. I think if a deployment like this is attractive for your application, you probably know it. I'm certainly not going to evangelize against people scaling up single-region databases!
Comments
Running a master in us-west and having read replica in europe, what could go wrong ...
My advice is don't do those things in the blog and keep the DB and the app in the same region it will save you so many problems. If you can't just use a DB that was designed for that like Spanner or Cockroach.
Genuine question: what could go wrong?
I'm a big fan of "pinning" mechanisms where any time a user performs a write to your database you set a cookie that lasts for a few seconds and causes any subsequent requests from that user to go to the lesser, not a replica.
This solves the stale-read-after-write problem and ensures users will see any changes that they have personally made.
This does assume a mostly-reads, occasional-write application (which defines most of the applications I've worked on in my career).
So given that kind of application, what are the problems I'm not considering that I should look out for with a read-replica on a different continent?
Of the many things that can go wrong, the one I would be most worried about would be replication lag causing the disk on the primary to fill with WAL logs bringing down the primary, or if you have limited the WAL logs, the replica not being able to keep up and detaching from the primary causing stale reads.
Also long distances will make replication latency increase so unless _all_ requests are sent to the primary region you will likely have a read after write state data situation unless you go out of your way to ensure consistency
Stale replicas are a "stop serving requests from a particular region" problem. We pretty aggressively monitor these and consider things broken if replication is lagged more than about 10s.
One thing about running thousands of DB clusters (which we did at Compose.com before this) is, these problems exist even without geo replication. And they exist at such a scale that you have to be good at handling them. Geo replicas are _more_ prone to these kinds of issues but we already had to be good at handling them.
That is good and should prevent against bringing down the primary. There is still the issue with being able to get that region back online once replication is severed, which isn't always possible even with co-located systems.
Is the WAL log situation you describe here a likely function of both distance (hence latency) and volume of writes? So if I have a high volume of writes AND I'm trying to replicate them half way around the globe I'm more at risk of running out of WAL log space on the leader?
Yeah, if you are doing global replication you are much more likely to fall behind so you should have aggressive monitoring on the replica to remove it if replica lag gets to high, that will prevent the primary from retaining WAL files for too long.
It also means you will need to be able to resync the replica from a backup once write volume calms down.
All the things that can go wrong with globally distributed databases. You can just run an HA cluster in a single region if you're looking to keep it simple, just the same way you can run a single-region app on Fly if you like.
We try to go out of our way to say that this stuff isn't a perfect fit for every application or a "best" way to do things. I think if a deployment like this is attractive for your application, you probably know it. I'm certainly not going to evangelize against people scaling up single-region databases!