A single node failure will be recovered from quickly without issue. Multiple concurrent failures are handled the same as the single failure case, so long as our failure assumptions are not violated (e.g., every node in the datacenter fails simultaneously).
That can work if the server process is killed so that the master is immediately notified. But what about other failure modes? For example, if the disk has soft errors and writes start taking several seconds to complete, the system can't decide in a small amount of time that the node is dead.
Comments
HyperDex uses value-dependent chaining, which offers fault tolerance properties similar to those provided by chain replication (http://www.cs.cornell.edu/home/rvr/papers/osdi04.pdf).
A single node failure will be recovered from quickly without issue. Multiple concurrent failures are handled the same as the single failure case, so long as our failure assumptions are not violated (e.g., every node in the datacenter fails simultaneously).
That can work if the server process is killed so that the master is immediately notified. But what about other failure modes? For example, if the disk has soft errors and writes start taking several seconds to complete, the system can't decide in a small amount of time that the node is dead.
In this case,the node should report its own failure.
It must be able to do so, otherwise it will be deemed faulty by the entity it reports to.