Clever, but only useful if your “pre-DB-write” work is cheap. For example, I work at a company where part of what we do is matching riders to drivers. This can be very expensive, and a typical flow is:
1) Read current state of riders/drivers from DB (slightly expensive)
2) Solve a vehicle routing problem with the new rider request added to the current state (can be VERY expensive)
3) If there’s a good new solution, commit the changes that the VRP solutions suggest (this is the DB write, and it’s only slightly expensive)
The approach in this blog post would have us duplicating the most expensive thing our app does (step 2 above), for most requests - not good. Much more load on our system, and these already slow writes would take ~twice as long.
I’d hope Fly also lets you configure the load balancer - i.e. have a way to send certain requests to the “writer” nodes by default, vs. in a retry.
Rather than using their suggested "catch writes and throw an exception" mechanism, I would instead write my own fast application logic to identify if something is likely to be a write.
For most of the applications I build the HTTP verb is good enough for this - so I would add a tiny piece of Django middleware which looks for a POST to a non-primary region and sends fly-replay straight away at that point.
I guess where it often gets complex, even with RESTful APIs, is:
- Deletes, puts and patches are basically guaranteed to be writes
- Get is GENERALLY read only, but it often updates caches - and your cache may have similar concerns. Also, sometimes gets write to the DB, i.e. updating a “last seen/last activity” type field
- Posts are generally writes, but definitely not always. Any time you really need a body for a read (big request coming from a browser, or just too much structure in the args to encode in HTTP query params), that read will be a post
I like your approach of short-circuiting and immediately sending fly-replay, but instead of doing it by HTTP method, I’d probably do it by manually marking endpoints as write endpoints. And then I’d also have a catch-all similar to the blog post, based on DB errors, that both sends fly-replay and logs. And then keep track of that log, if it ever happens that’s a sign that you need to mark a new endpoint as a writer.
We follow this pattern as well with a CloudFront -> OpenResty/Nginx router for the different HTTP methods. I've often wondered by CloudFront can't route to custom origins based on request method. You have to do it with a Lambda@Edge function, which is annoying.
GraphQL might work OK though, because it differentiates queries from mutations - so you could have some early logic that says "if this POST request includes a mutation, replay against the leader - otherwise keep running against the replica".
It's true! We designed this specifically because it works with GraphQL. The exception->replay works by default, and it's easy to send early replay command on GraphQL mutations.
What is more you could just have your client send all mutations to one API and all queries to another. I suppose you could do that with REST as well by in Graphql the deliniation is very clear and there are only 2 cases you need to handle.
Comments
Clever, but only useful if your “pre-DB-write” work is cheap. For example, I work at a company where part of what we do is matching riders to drivers. This can be very expensive, and a typical flow is:
1) Read current state of riders/drivers from DB (slightly expensive)
2) Solve a vehicle routing problem with the new rider request added to the current state (can be VERY expensive)
3) If there’s a good new solution, commit the changes that the VRP solutions suggest (this is the DB write, and it’s only slightly expensive)
The approach in this blog post would have us duplicating the most expensive thing our app does (step 2 above), for most requests - not good. Much more load on our system, and these already slow writes would take ~twice as long.
I’d hope Fly also lets you configure the load balancer - i.e. have a way to send certain requests to the “writer” nodes by default, vs. in a retry.
Rather than using their suggested "catch writes and throw an exception" mechanism, I would instead write my own fast application logic to identify if something is likely to be a write.
For most of the applications I build the HTTP verb is good enough for this - so I would add a tiny piece of Django middleware which looks for a POST to a non-primary region and sends fly-replay straight away at that point.
I guess where it often gets complex, even with RESTful APIs, is:
- Deletes, puts and patches are basically guaranteed to be writes
- Get is GENERALLY read only, but it often updates caches - and your cache may have similar concerns. Also, sometimes gets write to the DB, i.e. updating a “last seen/last activity” type field
- Posts are generally writes, but definitely not always. Any time you really need a body for a read (big request coming from a browser, or just too much structure in the args to encode in HTTP query params), that read will be a post
I like your approach of short-circuiting and immediately sending fly-replay, but instead of doing it by HTTP method, I’d probably do it by manually marking endpoints as write endpoints. And then I’d also have a catch-all similar to the blog post, based on DB errors, that both sends fly-replay and logs. And then keep track of that log, if it ever happens that’s a sign that you need to mark a new endpoint as a writer.
This is a good way to do it. Catching errors lets us reliably ship a library that makes this work for almost everyone, but it's not right for all apps: https://github.com/soupedup/fly-rails/blob/main/lib/fly-rail...
We follow this pattern as well with a CloudFront -> OpenResty/Nginx router for the different HTTP methods. I've often wondered by CloudFront can't route to custom origins based on request method. You have to do it with a Lambda@Edge function, which is annoying.
Cries in GraphQL (where you generally POST, even for queries).
GraphQL might work OK though, because it differentiates queries from mutations - so you could have some early logic that says "if this POST request includes a mutation, replay against the leader - otherwise keep running against the replica".
It's true! We designed this specifically because it works with GraphQL. The exception->replay works by default, and it's easy to send early replay command on GraphQL mutations.
What is more you could just have your client send all mutations to one API and all queries to another. I suppose you could do that with REST as well by in Graphql the deliniation is very clear and there are only 2 cases you need to handle.
A standard QUERY verb should fix this.
https://www.w3.org/2012/ldp/wiki/Proposal_for_HTTP_QUERY_Ver...
would one do it in the custom db router itself instead?