I wish I'd had more space to write about the global orchestrator design, because it's fun.
The Fly Machines orchestrator goes through some trouble to keep the source of truth for each VM decentralized, owned by the physical it runs on. But there's still global state --- apps, organizations, services. That stuff is all on Postgres. Postgres keeps up with it just fine but I'd be lying if I didn't say we're always looking out the corner of our eyes on metrics.
The global state for Sprites is on object storage. Each organization gets a separate SQLite database, and that database is synchronized to object storage with Litestream.io (Lightstream is load bearing in a bunch of places here; solid as a rock for us).
I think people really still sleep on the "multiple SQLite database" backing store design.
I've been working on the orchestrator side with Elixir and Phoenix, so happy to continue the discussion for curious minds. One of the coolest things we can do is things like this in Elixir - from any node we can reach out to a sqlite db across the planet:
OrgTracker.with_repo(org_id, fn ->
repo.all(from sprite in "sprites", select: ...)
end)
That will find or place an Elixir process on the cluster and rpc the target node with our code. Placements can be sticky so they pin to a machine so we don't have to suck down the db every start, but we also balance out the load and handle failover of durable processes automatically. Combined with litestream, the result is distributed sqlite with failover while treating it essentially like a locally reachable sqlite db. Yes there is the speed of light to contend with, but by sending the execution across the wire rather than individual queries, we only ever pay a single hop to reach the process/sqlite.
We use dns_cluster, which ships with all phoenix apps. libcluster achieves the same, so whatever works. Ultimately dist erl just needs a way to reach the nodes and you call Node.connect/1 on a hostname and off to the races. It's similar to FLAME, in that the erlang VM allows sending functions over the wire as a regular transparently encoded/decoded datastructure, but in this case it's just simple built-in erlang erpc, ie `:erpc.call` underneath rather than FLAME where we are managing a pool of elastic nodes, then rpc'ing them.
Thanks for the reply. Summary: Here's this really cool thing we're doing! Oh, are you wrote something exotic for it? Nope, just regular Erlang goodness that's been around 20 years.
Cloudflare definitely isn't with their Durable Object design. As an infra developer/Django user myself, I know it's a pretty big paradigm shift to think in the 1 DB per user realm.
However, I do think it'll unlock "globally available" apps as a first class citizen moving forward. If Uber were made today, I suspect it'd follow the 1 DB per user and be deployed really close to the edge with providers like Fly.io, Koyeb, or Cloudflare.
Comments
I wish I'd had more space to write about the global orchestrator design, because it's fun.
The Fly Machines orchestrator goes through some trouble to keep the source of truth for each VM decentralized, owned by the physical it runs on. But there's still global state --- apps, organizations, services. That stuff is all on Postgres. Postgres keeps up with it just fine but I'd be lying if I didn't say we're always looking out the corner of our eyes on metrics.
The global state for Sprites is on object storage. Each organization gets a separate SQLite database, and that database is synchronized to object storage with Litestream.io (Lightstream is load bearing in a bunch of places here; solid as a rock for us).
I think people really still sleep on the "multiple SQLite database" backing store design.
I've been working on the orchestrator side with Elixir and Phoenix, so happy to continue the discussion for curious minds. One of the coolest things we can do is things like this in Elixir - from any node we can reach out to a sqlite db across the planet:
OrgTracker.with_repo(org_id, fn ->
end)That will find or place an Elixir process on the cluster and rpc the target node with our code. Placements can be sticky so they pin to a machine so we don't have to suck down the db every start, but we also balance out the load and handle failover of durable processes automatically. Combined with litestream, the result is distributed sqlite with failover while treating it essentially like a locally reachable sqlite db. Yes there is the speed of light to contend with, but by sending the execution across the wire rather than individual queries, we only ever pay a single hop to reach the process/sqlite.
(Disclaimer: Newbie checking but still curious).
Are you using libluster or Distributed Erlang to reach the clusters? Or just simple networking over the Fly network.
Is this similar to what you did with Flame? Or just a refinement of that idea.
We use dns_cluster, which ships with all phoenix apps. libcluster achieves the same, so whatever works. Ultimately dist erl just needs a way to reach the nodes and you call Node.connect/1 on a hostname and off to the races. It's similar to FLAME, in that the erlang VM allows sending functions over the wire as a regular transparently encoded/decoded datastructure, but in this case it's just simple built-in erlang erpc, ie `:erpc.call` underneath rather than FLAME where we are managing a pool of elastic nodes, then rpc'ing them.
Thanks for the reply. Summary: Here's this really cool thing we're doing! Oh, are you wrote something exotic for it? Nope, just regular Erlang goodness that's been around 20 years.
Cloudflare definitely isn't with their Durable Object design. As an infra developer/Django user myself, I know it's a pretty big paradigm shift to think in the 1 DB per user realm.
However, I do think it'll unlock "globally available" apps as a first class citizen moving forward. If Uber were made today, I suspect it'd follow the 1 DB per user and be deployed really close to the edge with providers like Fly.io, Koyeb, or Cloudflare.
For those who missed it, tptacek wrote TFA.