Surprisingly, I'm already encountering a second solution that involves storing data chunks on S3 — and this is all within the same week.
This is becoming popular. At Playcode, we built what we believe is a revolutionary file system for our Playcode Cloud (https://playcode.io/cloud), which enables the creation of full-stack web software. The FS built completely from scratch using Rust. We thought we were the smartest ones around and that nobody else had figured this out. But it turns out Databricks, Neon, and several others have as well.
The idea behind a *Bottomless File System* is really cool, and it works very well for us. Essentially, as described here:
- There is a *page server*
- A *Linux file system* split into chunks (let's call them chunks instead of pages)
- A *cache on NVMe*
- And of course, *object storage*, where everything is asynchronously synchronized
It works quite well, though it has its downsides.
One clear advantage is that NVMe drives have become expensive lately, while object storage remains cheap — so the benefits are undeniable. That said, latency is also a factor.
On top of that, uplink costs are rising. To run an object storage-backed file system, you need a very strong uplink with consistent speed — 1 Gbps is simply not enough. Ideally, you want *5 to 10 Gbps*, depending on the load.
We spend a lot of time optimizing and experimenting with different hosting providers — specifically bare metal hardware. The main challenges are:
- *Slow disks*
- *Slow uplink*
- And as it turns out, *object storage can be unreliable* — unless you're using S3
But AWS hardware is expensive, so nothing in life is ever that simple.
I also want to point out that all of this sounds fun and great — until the load kicks in and usage starts to grow.
That's when you start seeing:
- *Rate limits* from object storage
- *Dropped packets*
- *Hanging S3 requests*
- *Overloaded NVMe drives* — because it turns out they're nowhere near as fast as they seem
For example, we recently discovered that the read speed is 5 GB/s, but the *average write speed is only 400 MB/s* — not several gigabytes as expected. Surprise! Who would have thought that Bare Metal could ship such underwhelming drives?
And then there's the CPU — which is also easy to kill, for instance, if you're compressing chunks. And so on, and so on.
A lot of things surface once you're in *production usage*. On paper, of course, everything looked much simpler.
Comments
Surprisingly, I'm already encountering a second solution that involves storing data chunks on S3 — and this is all within the same week.
This is becoming popular. At Playcode, we built what we believe is a revolutionary file system for our Playcode Cloud (https://playcode.io/cloud), which enables the creation of full-stack web software. The FS built completely from scratch using Rust. We thought we were the smartest ones around and that nobody else had figured this out. But it turns out Databricks, Neon, and several others have as well.
The idea behind a *Bottomless File System* is really cool, and it works very well for us. Essentially, as described here:
- There is a *page server* - A *Linux file system* split into chunks (let's call them chunks instead of pages) - A *cache on NVMe* - And of course, *object storage*, where everything is asynchronously synchronized
It works quite well, though it has its downsides.
One clear advantage is that NVMe drives have become expensive lately, while object storage remains cheap — so the benefits are undeniable. That said, latency is also a factor.
On top of that, uplink costs are rising. To run an object storage-backed file system, you need a very strong uplink with consistent speed — 1 Gbps is simply not enough. Ideally, you want *5 to 10 Gbps*, depending on the load.
We spend a lot of time optimizing and experimenting with different hosting providers — specifically bare metal hardware. The main challenges are:
- *Slow disks* - *Slow uplink* - And as it turns out, *object storage can be unreliable* — unless you're using S3
But AWS hardware is expensive, so nothing in life is ever that simple.
I also want to point out that all of this sounds fun and great — until the load kicks in and usage starts to grow.
That's when you start seeing:
- *Rate limits* from object storage - *Dropped packets* - *Hanging S3 requests* - *Overloaded NVMe drives* — because it turns out they're nowhere near as fast as they seem
For example, we recently discovered that the read speed is 5 GB/s, but the *average write speed is only 400 MB/s* — not several gigabytes as expected. Surprise! Who would have thought that Bare Metal could ship such underwhelming drives?
And then there's the CPU — which is also easy to kill, for instance, if you're compressing chunks. And so on, and so on.
A lot of things surface once you're in *production usage*. On paper, of course, everything looked much simpler.