I've been kicking around an idea like this for a while. The train of thought that brought me there was the recognition of a distinct memory hierarchy in today's common distributed applications that parallels the one in your computer.
So your computer has memory banks that (as a rough first approximation) each get ~10x bigger, but with ~10x greater latency. The neat part though is that global consistency for writes happens with L1 and L2 working together, so pretty damn high up in the hierarchy. In contrast distributed applications typically at best will have a write through cache where global consistency happens all the day down at the actual data store. Exploring the idea of distributed MOESI where one client, because they had the permissions to write something in the first place, can then be the owner for that database row as it's still being flushed out seems like a great basis for a distributed system that might not even need the dedicated datastore at all anymore, but a sea of clients participating in coherency and replication. Albeit this has oodles of consistency and availability problems that very well might kill the whole concept, like how MOESI would absolutely fall over when trying to hotplug CPUs at arbitrary times.
Maybe not directly related but this reminds of a paper[1] by Microsoft Research where they replicate state with DMA (Direct Memory Access) across the network and battery backups that have just enough power to flush memory to disk if the power goes out. you might find it interesting
In contrast distributed applications typically at best will have a write through cache where global consistency happens all the day down at the actual data store. Exploring the idea of distributed MOESI where one client, because they had the permissions to write something in the first place, can then be the owner for that database row as it's still being flushed out seems like a great basis for a distributed system that might not even need the dedicated datastore at all anymore, but a sea of clients participating in coherency and replication.
Modern commercial distributed file systems do the type of caching you’re talking about. More generally, a Distributed Lock Manager [1] gets you halfway there. You’re right that a write-through cache is still necessary to achieve redundancy requirements, but especially in the context of file systems you can often coalesce local edits until the user explicitly issues stage equivalent of a flush operation, and only do the write-through then.
Comments
I've been kicking around an idea like this for a while. The train of thought that brought me there was the recognition of a distinct memory hierarchy in today's common distributed applications that parallels the one in your computer.
So your computer has memory banks that (as a rough first approximation) each get ~10x bigger, but with ~10x greater latency. The neat part though is that global consistency for writes happens with L1 and L2 working together, so pretty damn high up in the hierarchy. In contrast distributed applications typically at best will have a write through cache where global consistency happens all the day down at the actual data store. Exploring the idea of distributed MOESI where one client, because they had the permissions to write something in the first place, can then be the owner for that database row as it's still being flushed out seems like a great basis for a distributed system that might not even need the dedicated datastore at all anymore, but a sea of clients participating in coherency and replication. Albeit this has oodles of consistency and availability problems that very well might kill the whole concept, like how MOESI would absolutely fall over when trying to hotplug CPUs at arbitrary times.
Maybe not directly related but this reminds of a paper[1] by Microsoft Research where they replicate state with DMA (Direct Memory Access) across the network and battery backups that have just enough power to flush memory to disk if the power goes out. you might find it interesting
[1] https://pdos.csail.mit.edu/6.824/papers/farm-2015.pdf
That's a fantastic paper, thanks for sharing.
Modern commercial distributed file systems do the type of caching you’re talking about. More generally, a Distributed Lock Manager [1] gets you halfway there. You’re right that a write-through cache is still necessary to achieve redundancy requirements, but especially in the context of file systems you can often coalesce local edits until the user explicitly issues stage equivalent of a flush operation, and only do the write-through then.
[1] https://en.m.wikipedia.org/wiki/Distributed_lock_manager
Some HSM ( https://en.wikipedia.org/wiki/Hierarchical_storage_managemen... ) tools and know-how may be pertinent.
For sure. I actually used to write tape library robotics firmware, so that theme of exploring memory hierarchies is absolutely on my mind.