We don't use ZFS (in theory, we give Fly Machines a raw block device), but I'd be interested in the sketch of how you'd do something like this in an end-to-end ZFS system, just so I can see how well I communicated the properties of our system in this post.
I was thinking zfs as the base filesystem with zvols for the Fly Machine so that would be you raw block device. I'm not sure how send/recv handles the trimming problem you outlined though
zvols on linux have known performance problems, see https://github.com/openzfs/zfs/issues/7631 -- most VM disk images on ZoL end up being regular files, for this reason.
zfs send+receive of those such work just fine, but the bigger problem is that zfs replication doesn't know about the "pull on demand, migrate in the background" design at all, so you're probably better not using it for migration.
Speaking as someone that has built experiments on top of zfs zvols (https://xeiaso.net/blog/waifud-progress-report-2/), I'm not entirely sure what the main usecase of them is. They seem to really hate journaled filesystems (nearly all of the filesystems currently in use). It's also really easy to run into cases where LVM inside a zvol for a VM makes the host kernel eagerly pick it up and pin it as active on reboot, preventing you from deleting the zvols in question (this took me a week to properly diagnose).
They probably made sense back when filesystems weren't journaled, but we don't live in that world anymore.
Comments
My initial thought was that ZFS replication would be excellent for this but I guess it is not low level enough?
We don't use ZFS (in theory, we give Fly Machines a raw block device), but I'd be interested in the sketch of how you'd do something like this in an end-to-end ZFS system, just so I can see how well I communicated the properties of our system in this post.
I was thinking zfs as the base filesystem with zvols for the Fly Machine so that would be you raw block device. I'm not sure how send/recv handles the trimming problem you outlined though
zvols on linux have known performance problems, see https://github.com/openzfs/zfs/issues/7631 -- most VM disk images on ZoL end up being regular files, for this reason.
zfs send+receive of those such work just fine, but the bigger problem is that zfs replication doesn't know about the "pull on demand, migrate in the background" design at all, so you're probably better not using it for migration.
(I work for fly.io, but not on Machines)
Speaking as someone that has built experiments on top of zfs zvols (https://xeiaso.net/blog/waifud-progress-report-2/), I'm not entirely sure what the main usecase of them is. They seem to really hate journaled filesystems (nearly all of the filesystems currently in use). It's also really easy to run into cases where LVM inside a zvol for a VM makes the host kernel eagerly pick it up and pin it as active on reboot, preventing you from deleting the zvols in question (this took me a week to properly diagnose).
They probably made sense back when filesystems weren't journaled, but we don't live in that world anymore.