Wow. I thought of etcd as taking consistency very seriously. Aphyr's discoveries are indeed quite surprising: "The very first test I ran with reported a linearizability failure. I was so surprised I spent another week double-checking Knossos and Jepsen, then writing my own etcd client, to make sure I hadn’t made a mistake. Sure enough, etcd’s registers are not linearizable."
I'm not sure what you've added here other than to exclaim surprise. Bugs can be surprising, bugs are usually not intentional. Given the fast response by the etcd team, they seem to be taking consistency very seriously.
This was not a bug, it was a design decision (source: https://github.com/coreos/etcd/issues/741 ) . You can get stale data in some circumstances because they wanted to avoid the performance penalty of ensuring the latest really is the latest value from a quorum of followers.
There's the design decision, and then there's the incorrect documentation that says reads with "consistent=true" are guaranteed to return the latest value.
Quote from the wiki:
A history is linearizable if:
its invocations and responses can be reordered to yield a sequential history
that sequential history is correct according to the sequential definition of the object
if a response preceded an invocation in the original history, it must still precede it in the sequential reordering.
Please forgive my stupidness. But could you tell me which one it violates?
It is possible that the transaction log's last entry contains a value that is not yet considered 'committed' at large, as per the paper. This transaction needs to be confirmed later on when an additional transaction comes and supercede it, confirming it was committed by all.
It is also possible for a leader to be demoted during a split, where the log of that partial transaction will not be counted as final. The new leader at this point can then force a truncation of a follower's log, or ignore it entirely.
The entry you have read from a node that was thought to still be a master without first consulting a majority is therefore possibly a bad write that won't be part of history as far as consensus goes.
This is explained later in the original Raft paper, and this is why you need to read from the quorum to be able to guarantee consistency under all circumstances, among other problem cases.
Comments
Wow. I thought of etcd as taking consistency very seriously. Aphyr's discoveries are indeed quite surprising: "The very first test I ran with reported a linearizability failure. I was so surprised I spent another week double-checking Knossos and Jepsen, then writing my own etcd client, to make sure I hadn’t made a mistake. Sure enough, etcd’s registers are not linearizable."
I'm not sure what you've added here other than to exclaim surprise. Bugs can be surprising, bugs are usually not intentional. Given the fast response by the etcd team, they seem to be taking consistency very seriously.
This was not a bug, it was a design decision (source: https://github.com/coreos/etcd/issues/741 ) . You can get stale data in some circumstances because they wanted to avoid the performance penalty of ensuring the latest really is the latest value from a quorum of followers.
There's the design decision, and then there's the incorrect documentation that says reads with "consistent=true" are guaranteed to return the latest value.
https://github.com/coreos/etcd/blob/master/Documentation/api...
etcd does provide linearizability with regard to the logical clock index.
What you found is not a bug, but a design decision.
Maybe the coreos people just should not assume that linearizability means the same thing to all people. And they should document it clearly.
Quote from the wiki: A history is linearizable if:
Please forgive my stupidness. But could you tell me which one it violates?It is possible that the transaction log's last entry contains a value that is not yet considered 'committed' at large, as per the paper. This transaction needs to be confirmed later on when an additional transaction comes and supercede it, confirming it was committed by all.
It is also possible for a leader to be demoted during a split, where the log of that partial transaction will not be counted as final. The new leader at this point can then force a truncation of a follower's log, or ignore it entirely.
The entry you have read from a node that was thought to still be a master without first consulting a majority is therefore possibly a bad write that won't be part of history as far as consensus goes.
This is explained later in the original Raft paper, and this is why you need to read from the quorum to be able to guarantee consistency under all circumstances, among other problem cases.