I'm the author of a distributed database, competing with them. Overall I'm refreshingly impressed with their design document. Which I am sad to say that most other databases don't come any where near thinking these things through except as an after thought - so I am glad to see they are making it their priority.
With that said, they seem to be assuming that their clock skew (ε) has a fixed maximum boundary which is incredibly disconcerting to me as it implies that in certain (rare and anomalous) network partitions that they'll get data corruption and fail.
I can see how they, coming from a Spanner background with atomic clocks, might assume this. But this assumption requires that their database cluster is always connected, within some heartbeat interval (which they mention) such that they can trust there exists a maximum bounded ε skew.
So while it seems like a dumb question, I honestly must ask a very trivial question: how does CockroachDB handle basic network partitions? I assume they have a good answer to this, but it needs to be clarified in order to answer the more important issue of anomalous partitions, like split brain. This might rip the crockroach in half, quite literally, meaning that all other "guarantees" they give get thrown out the window like linearizability and global consistency.
Cockroach trusts the MaxOffset, and if your clocks don't live up to the promise, you might get some stale reads. By the way, Spanner breaks in the same way if their clock offset (via their TrueTime API) fails them. But Spanner has to wait out the MaxOffset on every commit, we don't - so we get away with having it high enough for off-the-shelf clock synchronization and save you the atomic clocks, at similar guarantees. That's a very good deal. If you happen to have atomic clocks around and you have strong guarantees on your uncertainty like Spanner does, you get linearizability at the same price.
Just from skimming the design doc, it appears that if your clock skew exceeds the maximum bound, it would break linearizability. I haven't parsed through all the details of their SSI implementation, but it appears that even with arbitrary skew they would still enforce serializable transactions. However, it appears that performance under high skew would drop off dramatically.
Without a global clock you basically have to give up uncontended snapshot reads and linearizability for cross-shard transactions. That would be a completely different system from spanner and cockroachdb.
I believe that if a node in the consensus group exceeds ε clock skew, it will be kicked out of the group.
As far as network partitions go, a consensus must exist for reads or writes. If you don't have 3 out of 5 working correctly and talking to each other, then you are down.
That's correct, it's a consistent system and so the majority needs to be involved on mutating writes. Reads typically can read from one designated copy of the replica directly (bypassing Raft).
Comments
I'm the author of a distributed database, competing with them. Overall I'm refreshingly impressed with their design document. Which I am sad to say that most other databases don't come any where near thinking these things through except as an after thought - so I am glad to see they are making it their priority.
With that said, they seem to be assuming that their clock skew (ε) has a fixed maximum boundary which is incredibly disconcerting to me as it implies that in certain (rare and anomalous) network partitions that they'll get data corruption and fail.
I can see how they, coming from a Spanner background with atomic clocks, might assume this. But this assumption requires that their database cluster is always connected, within some heartbeat interval (which they mention) such that they can trust there exists a maximum bounded ε skew.
So while it seems like a dumb question, I honestly must ask a very trivial question: how does CockroachDB handle basic network partitions? I assume they have a good answer to this, but it needs to be clarified in order to answer the more important issue of anomalous partitions, like split brain. This might rip the crockroach in half, quite literally, meaning that all other "guarantees" they give get thrown out the window like linearizability and global consistency.
Cockroach trusts the MaxOffset, and if your clocks don't live up to the promise, you might get some stale reads. By the way, Spanner breaks in the same way if their clock offset (via their TrueTime API) fails them. But Spanner has to wait out the MaxOffset on every commit, we don't - so we get away with having it high enough for off-the-shelf clock synchronization and save you the atomic clocks, at similar guarantees. That's a very good deal. If you happen to have atomic clocks around and you have strong guarantees on your uncertainty like Spanner does, you get linearizability at the same price.
For a more in-depth explanation of the above, see https://gist.github.com/tschottdorf/57bcccc379b151456044.
Just from skimming the design doc, it appears that if your clock skew exceeds the maximum bound, it would break linearizability. I haven't parsed through all the details of their SSI implementation, but it appears that even with arbitrary skew they would still enforce serializable transactions. However, it appears that performance under high skew would drop off dramatically.
Without a global clock you basically have to give up uncontended snapshot reads and linearizability for cross-shard transactions. That would be a completely different system from spanner and cockroachdb.
I believe that if a node in the consensus group exceeds ε clock skew, it will be kicked out of the group.
As far as network partitions go, a consensus must exist for reads or writes. If you don't have 3 out of 5 working correctly and talking to each other, then you are down.
That's correct, it's a consistent system and so the majority needs to be involved on mutating writes. Reads typically can read from one designated copy of the replica directly (bypassing Raft).