Comment on Redis latency spikes and the Linux kernel: a few more detailsparentComments−seiji11ySee the "How it works" section under "Snapshotting" at http://redis.io/topics/persistence−Ono-Sendai11yOk. A couple of points - you can do this kind of persistence with an immutable data structure without forking.Secondly, huge pages help to reduce memory latency for large data sets quite significantly.−seiji11yyou can do this kind of persistence with an immutable data structure without forkingHow?huge pages help to reduce memory latencyYup, but this is about _transparent_ huge pages which break databases all over the place: https://blogs.oracle.com/linux/entry/performance_issues_with... and http://dev.nuodb.com/techblog/linux-transparent-huge-pages-j... and http://scn.sap.com/people/markmumy/blog/2014/05/22/sap-iq-an... and http://www.percona.com/blog/2014/07/23/why-tokudb-hates-tran... and (a dozen more including varnish, mongo, hadoop, ...)−Ono-Sendai11yIf you have a pointer to an immutable data structure, you know it won't change, so you can write it out to disk at your leisure.−justincormack11yBut the immutable data structure is not that much different from the copy-on-write pages the kernel gives you, likely to involve the same amount of copying. And disk IO is blocking so you need at least a thread, so it is a natural strategy.
Comments
See the "How it works" section under "Snapshotting" at http://redis.io/topics/persistence
Ok. A couple of points - you can do this kind of persistence with an immutable data structure without forking.
Secondly, huge pages help to reduce memory latency for large data sets quite significantly.
you can do this kind of persistence with an immutable data structure without forking
How?
huge pages help to reduce memory latency
Yup, but this is about _transparent_ huge pages which break databases all over the place: https://blogs.oracle.com/linux/entry/performance_issues_with... and http://dev.nuodb.com/techblog/linux-transparent-huge-pages-j... and http://scn.sap.com/people/markmumy/blog/2014/05/22/sap-iq-an... and http://www.percona.com/blog/2014/07/23/why-tokudb-hates-tran... and (a dozen more including varnish, mongo, hadoop, ...)
If you have a pointer to an immutable data structure, you know it won't change, so you can write it out to disk at your leisure.
But the immutable data structure is not that much different from the copy-on-write pages the kernel gives you, likely to involve the same amount of copying. And disk IO is blocking so you need at least a thread, so it is a natural strategy.