The way this is implemented, with the ability for an application server attached to a replica to say "error: this needs to perform a write - hey CDN, replay this request against the region with the database leader in it" is SO clever.
I think most database engineers who think about ACID/isolation levels/etc would look at this and go "uhm... jumps out a window" but for a subset of users I guess it kind of works?
I don't really see how a transaction that performs reads and writes can be replayed like this with any sort of guarantee about anything, though.
And there's no opportunity to build any sort of conflict resolution in there because the replay is automatic.
The replay is only automatic if you opt for the version where you catch write errors and turn them into replay requests.
My experience of building web apps, where most business logic runs within the scope of a single HTTP request, suggests that a quite impressive number of common cases could be served by the replay-request-against-the-leader pattern.
I mean: just to be clear: you can't write to a read replica. There's no way to introduce a conflict that way.
Also just for what it's worth: we agree with 'foobarbazetc. There's a section about that in the post. If you're saying there are important classes of applications this doesn't work well for, that's true.
Just to be clear, I'm not trying to say it's bad or whatever. It's super cool! I'm a big fan of fly.io.
I just don't see how something like this actually works in a way where you can reason about transaction ordering without global serialisation of requests against the writable database, which I presume isn't happening behind the scenes?
Basically, this would seem to work great if every request did INSERTs only and you based no logic to run INSERTs on anything you SELECT'ed from the read only replicas.
But you could have situations where, e.g., you DELETE or UPDATE something in a request to one region, and it goes to replay that against the writable region, and in "replay gap" another request modifies the same rows or objects, and based on various factors such as latency etc, a DELETE ... WHERE or UPDATE ... WHERE clause might no longer hold. Or you UPDATE the wrong objects, or DELETE the wrong data, etc.
I did read the post, but I guess I need to re-read it. I have written a globally distributed database before so I'm always intrigued by how these work. :)
INSERT, DELETE, UPDATE will all fail in the readonly regions. This is normal for Postgres read replicas, we're not doing anything special here.
What we're doing works identically to a vanilla HTTP based app. Requests that modify the DB always run against the primary database. Requests that perform reads _then_ modify the DB always run against the primary database.
HTTP services all have an underlying eventual consistency problem. If you are viewing a page with a database ID on it, then click delete for that ID, the record could be gone before the time the request hits the database (because someone else might've clicked while you were reading).
Does that help? I think we didn't describe this well enough, it's way simpler under the covers than you might expect.
I don't think you're quite understanding how the replay mechanism works. The original HTTP request fails because it can't DELETE or UPDATE something, so the entire request gets re-routed and then handled by the app again, from the beginning. So your SELECTs (as long as they're in the same HTTP request) will always get run against the writable leader.
Think of it like, if your read-only regions don't have anything else stateful they're able to do before the postgres write fails, then their existence is indistinguishable from network latency. An app that doesn't break because of latency won't break because of this.
If the readonlies might do something else stateful and not unwind it on postgres write error, then this approach won't work. (But wouldn't they be buggy anyway? Postgres writes aren't guaranteed not to raise errors.)
That's because the database fail with no changes made, and they retry the whole HTTP request with all business logic in the application re-run against primary instance. So, nothing to do with database-level stuff.
The idea is basically "there is an useful concept of 'group of queries' which is invisible to DB" :)
That's no different than just having a single region serving your application with users that are different distances and latencies away. The one farther away can issue an UPDATE/DELETE at the same time as the closer user, and the closer one would win.
It's all a game of latencies and eventual consistency, so what Fly.io is just treating writes as an HTTP request that has to travel all the way to the primary original, triggered by error an the regional location rather than your application actively doing it.
Clever, but only useful if your “pre-DB-write” work is cheap. For example, I work at a company where part of what we do is matching riders to drivers. This can be very expensive, and a typical flow is:
1) Read current state of riders/drivers from DB (slightly expensive)
2) Solve a vehicle routing problem with the new rider request added to the current state (can be VERY expensive)
3) If there’s a good new solution, commit the changes that the VRP solutions suggest (this is the DB write, and it’s only slightly expensive)
The approach in this blog post would have us duplicating the most expensive thing our app does (step 2 above), for most requests - not good. Much more load on our system, and these already slow writes would take ~twice as long.
I’d hope Fly also lets you configure the load balancer - i.e. have a way to send certain requests to the “writer” nodes by default, vs. in a retry.
Rather than using their suggested "catch writes and throw an exception" mechanism, I would instead write my own fast application logic to identify if something is likely to be a write.
For most of the applications I build the HTTP verb is good enough for this - so I would add a tiny piece of Django middleware which looks for a POST to a non-primary region and sends fly-replay straight away at that point.
I guess where it often gets complex, even with RESTful APIs, is:
- Deletes, puts and patches are basically guaranteed to be writes
- Get is GENERALLY read only, but it often updates caches - and your cache may have similar concerns. Also, sometimes gets write to the DB, i.e. updating a “last seen/last activity” type field
- Posts are generally writes, but definitely not always. Any time you really need a body for a read (big request coming from a browser, or just too much structure in the args to encode in HTTP query params), that read will be a post
I like your approach of short-circuiting and immediately sending fly-replay, but instead of doing it by HTTP method, I’d probably do it by manually marking endpoints as write endpoints. And then I’d also have a catch-all similar to the blog post, based on DB errors, that both sends fly-replay and logs. And then keep track of that log, if it ever happens that’s a sign that you need to mark a new endpoint as a writer.
We follow this pattern as well with a CloudFront -> OpenResty/Nginx router for the different HTTP methods. I've often wondered by CloudFront can't route to custom origins based on request method. You have to do it with a Lambda@Edge function, which is annoying.
GraphQL might work OK though, because it differentiates queries from mutations - so you could have some early logic that says "if this POST request includes a mutation, replay against the leader - otherwise keep running against the replica".
It's true! We designed this specifically because it works with GraphQL. The exception->replay works by default, and it's easy to send early replay command on GraphQL mutations.
What is more you could just have your client send all mutations to one API and all queries to another. I suppose you could do that with REST as well by in Graphql the deliniation is very clear and there are only 2 cases you need to handle.
I'm in SJC (San Jose / Silicon Valley) and https://fly-global-rails.fly.dev/regions/syd shows that my replay time to their region in Sydney Australia is between 25 and 45ms. That's pretty quick!
Us West - Australia round-trip for them is ~140ms. And ~50ms seems to be the physical limit unless I messed up the numbers somewhere?
Edit:
Syd-syd response time 138ms
Syd-lax response time 427ms, replay time 16ms
Ok, not sure what the replay time means in this case (extra latency from the cancellation?), but it's not the total cost and you really don't want to send those packets around the world :-)
simonw was referencing the replay overhead number. His request hits Los Angeles first, the rails app in lax says "go to Sydney", and the request gets replayed in syd.
The entire request to Sydney for me (from Chicago) takes 516ms, 15ms of that is "replay overhead".
We need to optimize this. Right now, the edge proxy handles the replay request. So if you hit SJC and a server in LAX handles the request (which is probably what's happening for you), you pay the SJC -> LAX latency penalty before the replay happens.
There's no reason we can't replay from LAX, though, just need to build it.
The try / repeat pattern inevitably needs to do a full round trip multiple times in different regions.
What would be awesome to have is something like this but built on top of envoy postgres proxy or similar where it would know where to send the query to based on the table + column value / pk. But one then rebuilds Yugabyte.
my replay time to their region in Sydney Australia is between 25 and 45ms. That's pretty quick!
What on earth.. Are people really OK with adding 25-45ms latency because they can't be bothered to route their query to a database that can actually service it?
So far, yes they're ok with that. We can also optimize most of that number away, down to <10ms for almost every app.
They're ok with it because a (valuable) write request that's 10% slower typically comes after a read request that's much, much faster. In this scenario:
my replay time to their region in Sydney Australia is between 25 and 45ms. That's pretty quick!
The first read only requests were probably <100ms for simonw. The replayed request (that he initiated with a click) was probably 500ms. When a request already takes 500ms, an additional 50ms of latency is basically noise.
The alternative is (often) for them to refactor their apps and then enforce write restrictions on code to make sure random GETs don't write to the DB unnecessarily.
Our goal was to make this work without requiring hairy code changes. People who are ready to do the work to optimize their apps can make things faster and use what we're doing now as a fallback, if they want.
When a request already takes 500ms, an additional 50ms of latency is basically noise.
I guess it really just depends on what you're trying to do. I would never consider 50ms as noise. It's an accumulation of various things taking 50ms that results in your request taking 500ms in the first place, which is very slow.
I'm interested in this though. I think it's a cool experiment!
Oh yes, that is a fair point. I'm specifically talking about the type of requests that always take 500ms. Almost every HTTP request from Australia to the US takes ~500ms.
This is meant to decrease the unfixable latency, it's probably not a good way to solve app level latency issues.
I think it could be done better by marking requests as either queries or mutations like GraphQL does, then you can just check which it is and forward accordingly.
Although even with REST you should be able to do the same thing, assume GETS can go to the read replicas, should work if you aren’t doing anything weird.
It didn't work cross region because no one builds an app expecting latency between their app server code and database. What we'd see on write requests was something like:
1. Query for data from read replica, perhaps for validation (0ms)
2. Do a write to the primary in a different region (20-400ms)
3. Query primary for consistency (20-400ms)
4. More queries against primary for consistency (20-400ms)
5. Maybe another write (20-400ms)
6. repeat
You can actually use our postgres this way if you want! But it breaks for most apps. It is much, much faster to ship the whole HTTP request where it needs to be than the move the database away from an app instance.
The problem is that most postgres clients can't handle multiple parametric queries in a single connection (necessary for them to be in a single transaction).
This came to light for me as I was trying to build an app on CockroachDB, and the cluster nature of that (at least for geo-redundant setups) implies a non-negligible latency in the low single digits.
Comments
The way this is implemented, with the ability for an application server attached to a replica to say "error: this needs to perform a write - hey CDN, replay this request against the region with the database leader in it" is SO clever.
I think most database engineers who think about ACID/isolation levels/etc would look at this and go "uhm... jumps out a window" but for a subset of users I guess it kind of works?
I don't really see how a transaction that performs reads and writes can be replayed like this with any sort of guarantee about anything, though.
And there's no opportunity to build any sort of conflict resolution in there because the replay is automatic.
Maybe I missed something though?
The replay is only automatic if you opt for the version where you catch write errors and turn them into replay requests.
My experience of building web apps, where most business logic runs within the scope of a single HTTP request, suggests that a quite impressive number of common cases could be served by the replay-request-against-the-leader pattern.
I mean: just to be clear: you can't write to a read replica. There's no way to introduce a conflict that way.
Also just for what it's worth: we agree with 'foobarbazetc. There's a section about that in the post. If you're saying there are important classes of applications this doesn't work well for, that's true.
Just to be clear, I'm not trying to say it's bad or whatever. It's super cool! I'm a big fan of fly.io.
I just don't see how something like this actually works in a way where you can reason about transaction ordering without global serialisation of requests against the writable database, which I presume isn't happening behind the scenes?
Basically, this would seem to work great if every request did INSERTs only and you based no logic to run INSERTs on anything you SELECT'ed from the read only replicas.
But you could have situations where, e.g., you DELETE or UPDATE something in a request to one region, and it goes to replay that against the writable region, and in "replay gap" another request modifies the same rows or objects, and based on various factors such as latency etc, a DELETE ... WHERE or UPDATE ... WHERE clause might no longer hold. Or you UPDATE the wrong objects, or DELETE the wrong data, etc.
I did read the post, but I guess I need to re-read it. I have written a globally distributed database before so I'm always intrigued by how these work. :)
INSERT, DELETE, UPDATE will all fail in the readonly regions. This is normal for Postgres read replicas, we're not doing anything special here.
What we're doing works identically to a vanilla HTTP based app. Requests that modify the DB always run against the primary database. Requests that perform reads _then_ modify the DB always run against the primary database.
HTTP services all have an underlying eventual consistency problem. If you are viewing a page with a database ID on it, then click delete for that ID, the record could be gone before the time the request hits the database (because someone else might've clicked while you were reading).
Does that help? I think we didn't describe this well enough, it's way simpler under the covers than you might expect.
I don't think you're quite understanding how the replay mechanism works. The original HTTP request fails because it can't DELETE or UPDATE something, so the entire request gets re-routed and then handled by the app again, from the beginning. So your SELECTs (as long as they're in the same HTTP request) will always get run against the writable leader.
Think of it like, if your read-only regions don't have anything else stateful they're able to do before the postgres write fails, then their existence is indistinguishable from network latency. An app that doesn't break because of latency won't break because of this.
If the readonlies might do something else stateful and not unwind it on postgres write error, then this approach won't work. (But wouldn't they be buggy anyway? Postgres writes aren't guaranteed not to raise errors.)
That's because the database fail with no changes made, and they retry the whole HTTP request with all business logic in the application re-run against primary instance. So, nothing to do with database-level stuff.
The idea is basically "there is an useful concept of 'group of queries' which is invisible to DB" :)
That's no different than just having a single region serving your application with users that are different distances and latencies away. The one farther away can issue an UPDATE/DELETE at the same time as the closer user, and the closer one would win.
It's all a game of latencies and eventual consistency, so what Fly.io is just treating writes as an HTTP request that has to travel all the way to the primary original, triggered by error an the regional location rather than your application actively doing it.
Clever, but only useful if your “pre-DB-write” work is cheap. For example, I work at a company where part of what we do is matching riders to drivers. This can be very expensive, and a typical flow is:
1) Read current state of riders/drivers from DB (slightly expensive)
2) Solve a vehicle routing problem with the new rider request added to the current state (can be VERY expensive)
3) If there’s a good new solution, commit the changes that the VRP solutions suggest (this is the DB write, and it’s only slightly expensive)
The approach in this blog post would have us duplicating the most expensive thing our app does (step 2 above), for most requests - not good. Much more load on our system, and these already slow writes would take ~twice as long.
I’d hope Fly also lets you configure the load balancer - i.e. have a way to send certain requests to the “writer” nodes by default, vs. in a retry.
Rather than using their suggested "catch writes and throw an exception" mechanism, I would instead write my own fast application logic to identify if something is likely to be a write.
For most of the applications I build the HTTP verb is good enough for this - so I would add a tiny piece of Django middleware which looks for a POST to a non-primary region and sends fly-replay straight away at that point.
I guess where it often gets complex, even with RESTful APIs, is:
- Deletes, puts and patches are basically guaranteed to be writes
- Get is GENERALLY read only, but it often updates caches - and your cache may have similar concerns. Also, sometimes gets write to the DB, i.e. updating a “last seen/last activity” type field
- Posts are generally writes, but definitely not always. Any time you really need a body for a read (big request coming from a browser, or just too much structure in the args to encode in HTTP query params), that read will be a post
I like your approach of short-circuiting and immediately sending fly-replay, but instead of doing it by HTTP method, I’d probably do it by manually marking endpoints as write endpoints. And then I’d also have a catch-all similar to the blog post, based on DB errors, that both sends fly-replay and logs. And then keep track of that log, if it ever happens that’s a sign that you need to mark a new endpoint as a writer.
This is a good way to do it. Catching errors lets us reliably ship a library that makes this work for almost everyone, but it's not right for all apps: https://github.com/soupedup/fly-rails/blob/main/lib/fly-rail...
We follow this pattern as well with a CloudFront -> OpenResty/Nginx router for the different HTTP methods. I've often wondered by CloudFront can't route to custom origins based on request method. You have to do it with a Lambda@Edge function, which is annoying.
Cries in GraphQL (where you generally POST, even for queries).
GraphQL might work OK though, because it differentiates queries from mutations - so you could have some early logic that says "if this POST request includes a mutation, replay against the leader - otherwise keep running against the replica".
It's true! We designed this specifically because it works with GraphQL. The exception->replay works by default, and it's easy to send early replay command on GraphQL mutations.
What is more you could just have your client send all mutations to one API and all queries to another. I suppose you could do that with REST as well by in Graphql the deliniation is very clear and there are only 2 cases you need to handle.
A standard QUERY verb should fix this.
https://www.w3.org/2012/ldp/wiki/Proposal_for_HTTP_QUERY_Ver...
would one do it in the custom db router itself instead?
And probably adds quite a lot of latency under load.
The demo at https://fly-global-rails.fly.dev/ lets you see replay times.
I'm in SJC (San Jose / Silicon Valley) and https://fly-global-rails.fly.dev/regions/syd shows that my replay time to their region in Sydney Australia is between 25 and 45ms. That's pretty quick!
That's not just quick, that's suspiciously quick. Here's a round-trip chart for Azure: https://docs.microsoft.com/en-us/azure/networking/media/azur...
Us West - Australia round-trip for them is ~140ms. And ~50ms seems to be the physical limit unless I messed up the numbers somewhere?
Edit:
Syd-syd response time 138ms
Syd-lax response time 427ms, replay time 16ms
Ok, not sure what the replay time means in this case (extra latency from the cancellation?), but it's not the total cost and you really don't want to send those packets around the world :-)
simonw was referencing the replay overhead number. His request hits Los Angeles first, the rails app in lax says "go to Sydney", and the request gets replayed in syd.
The entire request to Sydney for me (from Chicago) takes 516ms, 15ms of that is "replay overhead".
We need to optimize this. Right now, the edge proxy handles the replay request. So if you hit SJC and a server in LAX handles the request (which is probably what's happening for you), you pay the SJC -> LAX latency penalty before the replay happens.
There's no reason we can't replay from LAX, though, just need to build it.
The try / repeat pattern inevitably needs to do a full round trip multiple times in different regions.
What would be awesome to have is something like this but built on top of envoy postgres proxy or similar where it would know where to send the query to based on the table + column value / pk. But one then rebuilds Yugabyte.
That sounds like sharding, which Postgres supports already.
It does, doesn’t it? Every database can be sharded. But you have to do it yourself. This is how it works in Yugabyte: https://docs.yugabyte.com/latest/explore/multi-region-deploy....
What on earth.. Are people really OK with adding 25-45ms latency because they can't be bothered to route their query to a database that can actually service it?
So far, yes they're ok with that. We can also optimize most of that number away, down to <10ms for almost every app.
They're ok with it because a (valuable) write request that's 10% slower typically comes after a read request that's much, much faster. In this scenario:
The first read only requests were probably <100ms for simonw. The replayed request (that he initiated with a click) was probably 500ms. When a request already takes 500ms, an additional 50ms of latency is basically noise.
The alternative is (often) for them to refactor their apps and then enforce write restrictions on code to make sure random GETs don't write to the DB unnecessarily.
Our goal was to make this work without requiring hairy code changes. People who are ready to do the work to optimize their apps can make things faster and use what we're doing now as a fallback, if they want.
I guess it really just depends on what you're trying to do. I would never consider 50ms as noise. It's an accumulation of various things taking 50ms that results in your request taking 500ms in the first place, which is very slow.
I'm interested in this though. I think it's a cool experiment!
Oh yes, that is a fair point. I'm specifically talking about the type of requests that always take 500ms. Almost every HTTP request from Australia to the US takes ~500ms.
This is meant to decrease the unfixable latency, it's probably not a good way to solve app level latency issues.
My experience working in tech is that it's pretty common for teams to add 100ms latency without even noticing!
I wish that wasn't true, but that's what I've seen.
I think it could be done better by marking requests as either queries or mutations like GraphQL does, then you can just check which it is and forward accordingly.
Although even with REST you should be able to do the same thing, assume GETS can go to the read replicas, should work if you aren’t doing anything weird.
There's probably a way to do this in native postgres by abusing the FDW facility. No real need to involve the CDN or replay the request.
We tried running something like pgpool's split write setup, where the app or a proxy in front of the DB knows where to send transactions: https://www.pgpool.net/docs/latest/en/html/runtime-config-lo...
It didn't work cross region because no one builds an app expecting latency between their app server code and database. What we'd see on write requests was something like:
1. Query for data from read replica, perhaps for validation (0ms)
2. Do a write to the primary in a different region (20-400ms)
3. Query primary for consistency (20-400ms)
4. More queries against primary for consistency (20-400ms)
5. Maybe another write (20-400ms)
6. repeat
You can actually use our postgres this way if you want! But it breaks for most apps. It is much, much faster to ship the whole HTTP request where it needs to be than the move the database away from an app instance.
The problem is that most postgres clients can't handle multiple parametric queries in a single connection (necessary for them to be in a single transaction).
This came to light for me as I was trying to build an app on CockroachDB, and the cluster nature of that (at least for geo-redundant setups) implies a non-negligible latency in the low single digits.