I like the content of this post, but I take issue with the idea that the debate only matters in a failure condition. That's academically true, but practically misleading. When developing my app, I have to assume any part of the system could fail at any time. Thus my whole app needs to be written with failure in mind.
In an ACID system, that means my transactions either happen or don't, and it's on me to figure out whether they did or not. Sometimes I'll need to do some investigating after a failure to find out what succeeded and what rolled back.
In an eventually consistent system, I need a contingency plan for every operation on the system. Now that contingency plan might be very simple, such as "Whatever happened, happened and last writer wins." The CALM conjecture paper does a decent job of describing when this might be a workable contingency plan. More complicated operations require increasingly complex contingency plans. This is why Cassandra is now offering counters as an API feature, because something as simple as counters is really hard to get right in all failure scenarios.
If you fit the CALM description and require multi-data center availability, then EC is probably a good bet. It may still be a good bet otherwise, but the availability comes along with a much more difficult app development picture, failures or no.
> This is why Cassandra is now offering counters as an API feature
This is specific to Cassandra, where the development team chose not to implement optimistic locking via version vectors. That said the counters patch essentially implements a vector of counters, very similar conceptually to a vector clock.
That said, there are some scenarios where quorums (in absence of an agreed upon view of the cluster e.g., a zookeeper-based failure detectors) could mean concurrent vector clocks, which means it's up to the application to reconcile it.
If your application can not reconcile a split brain scenario on its own and depends on a total order then yes, it's not a good fit for EC.
As I've said, I also think that serializable consistency is easier for applications to deal with (they can assume a total order) than eventual consistency but provides the strong benefit of read availability (something, quite frankly, most Internet applications require).
Most complex applications have points where consistency can be relaxed (either somewhat or fully) and points where consistency is needed.
In "we chose consistency" and "we chose availability" cases, the applications will have to deal with the consequences of those choices: much as developers may handwave the implications of an eventually consistent system (since it returns right results most of the time, not thinking whether or not their application is sensitive to a non-serializable order) they might also implement ad-hoc, poorly thought out HA solutions on top of strong consistent systems -- often times losing both availability and consistency.
The typical usage scenario of masking MySQL failures and latency by using memcache comes to mind. VoltDB is very correct to remove the need to use memcached for read latency (a caching layer in front of a database which already a cache which sits on top of a file system with a cache is a caching layer too many), but in many cases I've seen memcached used to accept reads (and sometimes writes) while the underlying database is unavailable.
Finally, the whole debate about atomicity is quite meaningless if (like many applications) you're fronting your data access layer with an RPC framework (your RPC call may go through to a database and perform a write even it times out to the application i.e., implying non-atomicity).
The CALM paper is a big move into the right direction: being able to determine synchronization points at which ACID transactions (or weaker forms of synchronizations if permissible e.g., quorums and version vectors) can be used. Incidentally, I also think a declarative language (whether like Bloom/Bud or closer to SQL) is a good way to express the constraints and let a system (which may not be all that different from a query planner) determine whether those points occur.
As for RPC, whether the client is informed of success or failure has nothing to do with atomicity. The guarantee is that the transaction either happens completely or not at all, and nothing more.
Agreed, it can be very frustrating that under certain failure scenarios, it's unclear whether the transaction completed or rolled back. Still, given atomicity, a correctly designed system can't be left in an inconsistent state. At times, it's up to the application to discover (or re-discover) that state.
Systems that offer atomicity for single record operations can actually make the same assurances, but may require complicated escrow systems and compensating transactions.
Comments
I like the content of this post, but I take issue with the idea that the debate only matters in a failure condition. That's academically true, but practically misleading. When developing my app, I have to assume any part of the system could fail at any time. Thus my whole app needs to be written with failure in mind.
In an ACID system, that means my transactions either happen or don't, and it's on me to figure out whether they did or not. Sometimes I'll need to do some investigating after a failure to find out what succeeded and what rolled back.
In an eventually consistent system, I need a contingency plan for every operation on the system. Now that contingency plan might be very simple, such as "Whatever happened, happened and last writer wins." The CALM conjecture paper does a decent job of describing when this might be a workable contingency plan. More complicated operations require increasingly complex contingency plans. This is why Cassandra is now offering counters as an API feature, because something as simple as counters is really hard to get right in all failure scenarios.
If you fit the CALM description and require multi-data center availability, then EC is probably a good bet. It may still be a good bet otherwise, but the availability comes along with a much more difficult app development picture, failures or no.
> This is why Cassandra is now offering counters as an API feature
This is specific to Cassandra, where the development team chose not to implement optimistic locking via version vectors. That said the counters patch essentially implements a vector of counters, very similar conceptually to a vector clock.
That said, there are some scenarios where quorums (in absence of an agreed upon view of the cluster e.g., a zookeeper-based failure detectors) could mean concurrent vector clocks, which means it's up to the application to reconcile it.
If your application can not reconcile a split brain scenario on its own and depends on a total order then yes, it's not a good fit for EC.
As I've said, I also think that serializable consistency is easier for applications to deal with (they can assume a total order) than eventual consistency but provides the strong benefit of read availability (something, quite frankly, most Internet applications require).
Most complex applications have points where consistency can be relaxed (either somewhat or fully) and points where consistency is needed.
In "we chose consistency" and "we chose availability" cases, the applications will have to deal with the consequences of those choices: much as developers may handwave the implications of an eventually consistent system (since it returns right results most of the time, not thinking whether or not their application is sensitive to a non-serializable order) they might also implement ad-hoc, poorly thought out HA solutions on top of strong consistent systems -- often times losing both availability and consistency.
The typical usage scenario of masking MySQL failures and latency by using memcache comes to mind. VoltDB is very correct to remove the need to use memcached for read latency (a caching layer in front of a database which already a cache which sits on top of a file system with a cache is a caching layer too many), but in many cases I've seen memcached used to accept reads (and sometimes writes) while the underlying database is unavailable.
Finally, the whole debate about atomicity is quite meaningless if (like many applications) you're fronting your data access layer with an RPC framework (your RPC call may go through to a database and perform a write even it times out to the application i.e., implying non-atomicity).
The CALM paper is a big move into the right direction: being able to determine synchronization points at which ACID transactions (or weaker forms of synchronizations if permissible e.g., quorums and version vectors) can be used. Incidentally, I also think a declarative language (whether like Bloom/Bud or closer to SQL) is a good way to express the constraints and let a system (which may not be all that different from a query planner) determine whether those points occur.
Overall, another solid comment.
As for RPC, whether the client is informed of success or failure has nothing to do with atomicity. The guarantee is that the transaction either happens completely or not at all, and nothing more.
Agreed, it can be very frustrating that under certain failure scenarios, it's unclear whether the transaction completed or rolled back. Still, given atomicity, a correctly designed system can't be left in an inconsistent state. At times, it's up to the application to discover (or re-discover) that state.
Systems that offer atomicity for single record operations can actually make the same assurances, but may require complicated escrow systems and compensating transactions.