> Yes. (Technically, append-at-head and delete-from-tail only.)
Do you handle this by breaking the DB up into multiple small files, like BDB-JE?
> The code keeps track of how much garbage is present and keeps a running tally of how much garbage collection it "owes" based on maintaining a long-term optimal GC rate. When that value is large enough, it looks for some old pages to clean.
How long-term is that? I assume you have considered the degenerate case, where load is increasing linearly (instead of more common daily/weekly/etc. cycles) and garbage collection falls behind. In context of your next answer, it seems like this case would cause the GC to get starved out, leading to higher IO requirements for queries, leading to less time to GC, and so on to failure.
Granted, you would have to be running very hot for this to happen, but its possible.
One more question: Have you considered implementing in-memory locking/synchronisation (like a shared mutex)? Offering test-and-set type operations is a nice alternative to full-fledge transactions for many use cases. If these are just used as synchronisation primitives then fsyncing them every time seems wasteful, on the assumption that many use cases don't care about lock durability across server failure.
The block store component in kivaloo uses multiple files, yes.
Needing to do GC won't make you need more I/Os to service requests; the exact same sequence of B+Tree nodes will need to be loaded from disk. The only effect of GC is wasted disk space.
Kivaloo does support a data-loss mode, so you could store locks in a daemon running with that option. Personally I prefer to err on the side of caution when I'm dealing with locking and transactions.
Comments
Thanks for the answers, this is very interesting.
> Yes. (Technically, append-at-head and delete-from-tail only.)
Do you handle this by breaking the DB up into multiple small files, like BDB-JE?
> The code keeps track of how much garbage is present and keeps a running tally of how much garbage collection it "owes" based on maintaining a long-term optimal GC rate. When that value is large enough, it looks for some old pages to clean.
How long-term is that? I assume you have considered the degenerate case, where load is increasing linearly (instead of more common daily/weekly/etc. cycles) and garbage collection falls behind. In context of your next answer, it seems like this case would cause the GC to get starved out, leading to higher IO requirements for queries, leading to less time to GC, and so on to failure.
Granted, you would have to be running very hot for this to happen, but its possible.
One more question: Have you considered implementing in-memory locking/synchronisation (like a shared mutex)? Offering test-and-set type operations is a nice alternative to full-fledge transactions for many use cases. If these are just used as synchronisation primitives then fsyncing them every time seems wasteful, on the assumption that many use cases don't care about lock durability across server failure.
The block store component in kivaloo uses multiple files, yes.
Needing to do GC won't make you need more I/Os to service requests; the exact same sequence of B+Tree nodes will need to be loaded from disk. The only effect of GC is wasted disk space.
Kivaloo does support a data-loss mode, so you could store locks in a daemon running with that option. Personally I prefer to err on the side of caution when I'm dealing with locking and transactions.