I suspect you might want to have a quick look at the numbers in your first sentence and have a rethink ;-)
I can't remember whether Linux supported the kind of fancy disc migration we wanted to do back in 2011.
I don't think it did. There was one other important difference (reading between the lines) between fly and us, and that's that we weren't using the kernel's NBD implementation, at all. At the time it had a hard limit of (from memory) 10 nbd mounts, so it was out of the running for the number of VMs we wanted on each head. We had to do it in user space, which meant we never had the problem of kernel threads locking up when the network had a moment. It was flexnbd talking directly to qemu, which presented a virtio block device to the guest.
That meant not using LVM2 or any of the other niceties directly in bigv itself. We did use it for administering the systems that ran bigv, and we used it in other places, but from memory the head and tail code itself had no knowledge of volumes or partitions. From the tail's point of view the customer disks were just files on a filesystem, and as far as I'm concerned that was a massive win - it meant that grabbing someone's VM to debug it needed no tools other than being able to recover the file, which isn't the case on some other storage systems.
It also meant that I ended up doing some of the most fun code I've ever written, getting the multithreaded code in flexnbd right. Although I do remember commenting fairly early on that threads in C might, perhaps, not be the finest choice of concurrency primitive for that particular job :-)
It is particularly sweet trick to have a suspicion about a server and just say "flush it!" and in 12-24 hours, it's no longer in service.
Being able to do that with both the heads and the tails was nice, and it's fun seeing the same ideas we built crop up in other places. I'm not sure exactly how far ahead of the curve we were but it's definitely taken a while for some of these ideas to spread - largely, I suspect, because kubernetes came along and stomped all over the conceptual space, and it's just got different ideas about how a lot of this stuff works. I do know that no other nbd server at the time could do what we were doing correctly, because if it could have, we'd have used it.
That experience left me with a few lasting impressions:
1. Hardware RAID is a scam. Rebuilds take too long, more disks fail, the firmware is buggy, it's expensive. You're best off using the hardware as a convenient way to ram more disks into the chassis than using its RAID features.
2. You're lucky if you get 12 hours' notice. For storage volumes that size, where you're already on network storage, you want a snapshot to already be available off the machine when you make the decision to evacuate. And that means some synchronisation algorithm running all the time. There are systems around which do that now - Longhorn springs to mind - and that would have been the next thing to build if I'd convinced anyone it was worthwhile. I understand fly saying "Raft is too complex" but then the flip side is that you have to move all the data at the worst possible moment. Maybe the network speed to disk size ratios they're dealing with make it make sense?
3. Small teams. Small teams all the way. Fly are doing this with - depending on how you count - between two and four times as many hands on keyboards as we had (for an international product, where ours was in two DC's, so there's that). There's a break-even point for how much faster you go with more developers, and it's usually lower than ten. People often don't understand how fast a team that small can go because they've never seen it first-hand, but having since seen the other extreme... yeah. It might not have felt like it at the time, but we got a lot right in terms of enabling ourselves to move fast.
I did miss this kind of work in 2022 and interviewed for a support role at fly. I'm not sure how to take being rejected at the screener stage, I'm sure some of my former staff might be able to explain it :)
No comment >:-) Having just spent a spectacularly unproductive afternoon trying to wrangle some particularly recalcitrant AWS terraform into a cooperative shape, I can say that this kind of work still needs doing, though...
Comments
I suspect you might want to have a quick look at the numbers in your first sentence and have a rethink ;-)
I don't think it did. There was one other important difference (reading between the lines) between fly and us, and that's that we weren't using the kernel's NBD implementation, at all. At the time it had a hard limit of (from memory) 10 nbd mounts, so it was out of the running for the number of VMs we wanted on each head. We had to do it in user space, which meant we never had the problem of kernel threads locking up when the network had a moment. It was flexnbd talking directly to qemu, which presented a virtio block device to the guest.
That meant not using LVM2 or any of the other niceties directly in bigv itself. We did use it for administering the systems that ran bigv, and we used it in other places, but from memory the head and tail code itself had no knowledge of volumes or partitions. From the tail's point of view the customer disks were just files on a filesystem, and as far as I'm concerned that was a massive win - it meant that grabbing someone's VM to debug it needed no tools other than being able to recover the file, which isn't the case on some other storage systems.
It also meant that I ended up doing some of the most fun code I've ever written, getting the multithreaded code in flexnbd right. Although I do remember commenting fairly early on that threads in C might, perhaps, not be the finest choice of concurrency primitive for that particular job :-)
Being able to do that with both the heads and the tails was nice, and it's fun seeing the same ideas we built crop up in other places. I'm not sure exactly how far ahead of the curve we were but it's definitely taken a while for some of these ideas to spread - largely, I suspect, because kubernetes came along and stomped all over the conceptual space, and it's just got different ideas about how a lot of this stuff works. I do know that no other nbd server at the time could do what we were doing correctly, because if it could have, we'd have used it.
That experience left me with a few lasting impressions:
1. Hardware RAID is a scam. Rebuilds take too long, more disks fail, the firmware is buggy, it's expensive. You're best off using the hardware as a convenient way to ram more disks into the chassis than using its RAID features.
2. You're lucky if you get 12 hours' notice. For storage volumes that size, where you're already on network storage, you want a snapshot to already be available off the machine when you make the decision to evacuate. And that means some synchronisation algorithm running all the time. There are systems around which do that now - Longhorn springs to mind - and that would have been the next thing to build if I'd convinced anyone it was worthwhile. I understand fly saying "Raft is too complex" but then the flip side is that you have to move all the data at the worst possible moment. Maybe the network speed to disk size ratios they're dealing with make it make sense?
3. Small teams. Small teams all the way. Fly are doing this with - depending on how you count - between two and four times as many hands on keyboards as we had (for an international product, where ours was in two DC's, so there's that). There's a break-even point for how much faster you go with more developers, and it's usually lower than ten. People often don't understand how fast a team that small can go because they've never seen it first-hand, but having since seen the other extreme... yeah. It might not have felt like it at the time, but we got a lot right in terms of enabling ourselves to move fast.
No comment >:-) Having just spent a spectacularly unproductive afternoon trying to wrangle some particularly recalcitrant AWS terraform into a cooperative shape, I can say that this kind of work still needs doing, though...