Pardon my ignorance - but isn't this a problem that's existed in database software in general, and one that's been dealt with in the past?
This seems to be a similar problem to how to do replication while also serving requests - on one side you want to preserve consistency, on the other side you want to make a full snapshot. It shouldn't matter how long the replication process takes (i.e. how slow fork is), seconds or days, as long as the incoming changes aren't faster than the replication rate. In this case though, your replication target is the disk.
It seems like there should never have to be a full lock of everything.
The problem is fork() is a blocking system call, and the side effects of fork also cause some latency when accessing the COW memory.
there should never have to be a full lock of everything.
There's no software lock involved from the Redis side, but fork() behavior has the effect of a pseudo-lock (because the system call blocks the entire process from running until it returns).
Comments
Pardon my ignorance - but isn't this a problem that's existed in database software in general, and one that's been dealt with in the past?
This seems to be a similar problem to how to do replication while also serving requests - on one side you want to preserve consistency, on the other side you want to make a full snapshot. It shouldn't matter how long the replication process takes (i.e. how slow fork is), seconds or days, as long as the incoming changes aren't faster than the replication rate. In this case though, your replication target is the disk.
It seems like there should never have to be a full lock of everything.
(i.e. how slow fork is)
The problem is fork() is a blocking system call, and the side effects of fork also cause some latency when accessing the COW memory.
there should never have to be a full lock of everything.
There's no software lock involved from the Redis side, but fork() behavior has the effect of a pseudo-lock (because the system call blocks the entire process from running until it returns).
Thank you, this clears up the issue a bit more in my mind.