It can be much simpler if you're willing to write a couple lines of assembler: use an 8-byte file that contains one 64-bit counter, mmap() it, mlock() it, and use atomic CPU instructions to increment and read it.
Great question: it does not! It just removes a potential source of stalls.
If the page isn't resident in the page cache, the thread(s) executing the atomic increment will take a page fault and be blocked until the file data is read from storage. The latency of the fault might be significant, but the counter will remain accurate.
There's an important caveat I should have added: the counter file might be very stale after a power failure. If you care about the counter integrity, you have to msync(MS_SYNC) periodically, and that's expensive. It might actually stall all threads interacting with the page, depending on the filesystem; that used to be true but I'm not certain it still is (see https://lwn.net/Articles/486311/). Where writeback is allowed to race with writes, whether you would be guaranteed the 8-byte value written back wasn't "torn" without explicitly blocking increments while syncing is also an interesting question if DMA is involved...
Comments
It can be much simpler if you're willing to write a couple lines of assembler: use an 8-byte file that contains one 64-bit counter, mmap() it, mlock() it, and use atomic CPU instructions to increment and read it.
Genuine question: Does it have to be mlocked for this to work?
Great question: it does not! It just removes a potential source of stalls.
If the page isn't resident in the page cache, the thread(s) executing the atomic increment will take a page fault and be blocked until the file data is read from storage. The latency of the fault might be significant, but the counter will remain accurate.
There's an important caveat I should have added: the counter file might be very stale after a power failure. If you care about the counter integrity, you have to msync(MS_SYNC) periodically, and that's expensive. It might actually stall all threads interacting with the page, depending on the filesystem; that used to be true but I'm not certain it still is (see https://lwn.net/Articles/486311/). Where writeback is allowed to race with writes, whether you would be guaranteed the 8-byte value written back wasn't "torn" without explicitly blocking increments while syncing is also an interesting question if DMA is involved...
I am going to ask gpt 4 to implement this for me so I can see what it looks like. Amazing.
I have to go to dinner so this isn't finished, but this is most of the code to prove it works: https://gist.github.com/jcalvinowens/0d7a5c327d863fca7c84daa...
How did it go?