(All credit for the idea in this comment is due to @aphyr)
Basically because the transactions modified keys selected from a uniform distribution, the probability of contention was extremely low. AKA this workload is basically a data-parallel problem, somewhat lessening the impressiveness of the high throughput. Would be interesting to see it with a Zipfian distribution (or even better, a Biebermark [0])
Actually, @aphyr's analysis is wrong. In fact it's actually worse that he says. There is an exactly 0% chance of conflicts between two transactions that each only write 20 random keys (as they do in this test). This is perhaps counterintuitive, but, because there are no reads, any serialization order is possible and therefore there is no chance of conflict.
Equally wrong is his assumption that this has any bearing on the performance of FoundationDB. In fact FoundationDB will perform the same whether the conflict rate is high or low. This isn't to say that FoundationDB somehow cheats the laws of transaction conflicts, just that it has to do all the work in either case. There is no trick or cheat on this test--this same performance will hold on a variety of workloads with varying conflict rates as well as those including reads.
That's true: a large conflict rate would eat into your budget because clients would be retrying.
Of course most real world workloads are (hopefully!) no where near 90% conflicts. If you had a 90% transaction conflict rate you could expect FoundationDB performance to drop by about 30-60% due to retries (and, worse news, you would need to rethink your architecture).
because there are no reads, any serialization order is possible and therefore there is no chance of conflict
I don't understand how this is the case. If clients A and B try to write to the same key, it can be serialized as {A,B} or {B,A}, but in either case, there is some kind of conflict... no?
Nope. Not unless someone actually read that same key in their transaction. Then, with optimistic concurrency at least, you might get a conflict at commit time that basically says "Hey, someone changed that key you read in the meantime so your write might not be correct anymore".
Ah – I see what you mean now! If they are blind writes, not some kind of CAS operation (reading from and modifying the same key), then there is no conflict.
I thought you meant that clients A and B do their writes, and only if someone at a later time were to observe the value at that key, a conflict would be caused (Heisenberg-style).
Just to further clarify, the transactions (and therefore transaction conflict detection) are more flexible than simple CAS operations. If transaction A reads key K1, and then writes to key K2, it will be rejected as conflicting if transaction B writes to key K1 and is serialized as happening before transaction K1 is committed (even though no other transaction wrote to key K2).
Every transaction in FoundationDB is submitted as a collection of writes/mutations, but also contains records of all keys read (and the consistent version at which they were read). In this test, the transactions all have empty conflict ranges, and thus cannot conflict with other transactions (but we still have to check!).
To be sure, he revels in the discomfort of others too much. But he's a young kid just finding his voice, and his contributions to practical distributed database discussions are pretty much without parallel. Just ignore the bluster and you'll learn something new from kyle every day, even if it's about his buttcheeks.
Unfortunately this problem isn't specific to FoundationDB; the old "industry-standard" TPC-C benchmark has a similar low-contention design which has led to years of unrepresentative performance tuning and benchmarketing.
How is it unrepresentative? TPC-C involves several queries simulating new orders being entered, being updated, filled, paid, etc. It is very representative of the workload it is simulating. If you want to simulate a different workload, pick one of the other benchmarks instead of C.
Absolutely. Choosing the proper data model based on your access patterns is one of the best ways of keeping FDB performance high by reducing the likelihood of contention.
Comments
This is very impressive, however...
See this tweet by @aphyr: https://twitter.com/aphyr/status/542755074380791809
(All credit for the idea in this comment is due to @aphyr)
Basically because the transactions modified keys selected from a uniform distribution, the probability of contention was extremely low. AKA this workload is basically a data-parallel problem, somewhat lessening the impressiveness of the high throughput. Would be interesting to see it with a Zipfian distribution (or even better, a Biebermark [0])
[0] - http://smalldatum.blogspot.co.il/2014/04/biebermarks.html
Actually, @aphyr's analysis is wrong. In fact it's actually worse that he says. There is an exactly 0% chance of conflicts between two transactions that each only write 20 random keys (as they do in this test). This is perhaps counterintuitive, but, because there are no reads, any serialization order is possible and therefore there is no chance of conflict.
Equally wrong is his assumption that this has any bearing on the performance of FoundationDB. In fact FoundationDB will perform the same whether the conflict rate is high or low. This isn't to say that FoundationDB somehow cheats the laws of transaction conflicts, just that it has to do all the work in either case. There is no trick or cheat on this test--this same performance will hold on a variety of workloads with varying conflict rates as well as those including reads.
Your definition of performance is a little weird.
Presumably you will want to retry conflicting transactions, so you generally would not count them towards your throughput.
For example if I commit 100 transactions per second, and 90% of them return conflicts, I am only successfully committing 10 transactions per second.
That's true: a large conflict rate would eat into your budget because clients would be retrying.
Of course most real world workloads are (hopefully!) no where near 90% conflicts. If you had a 90% transaction conflict rate you could expect FoundationDB performance to drop by about 30-60% due to retries (and, worse news, you would need to rethink your architecture).
I'm not an expert in this, but:
because there are no reads, any serialization order is possible and therefore there is no chance of conflict
I don't understand how this is the case. If clients A and B try to write to the same key, it can be serialized as {A,B} or {B,A}, but in either case, there is some kind of conflict... no?
Nope. Not unless someone actually read that same key in their transaction. Then, with optimistic concurrency at least, you might get a conflict at commit time that basically says "Hey, someone changed that key you read in the meantime so your write might not be correct anymore".
Ah – I see what you mean now! If they are blind writes, not some kind of CAS operation (reading from and modifying the same key), then there is no conflict.
I thought you meant that clients A and B do their writes, and only if someone at a later time were to observe the value at that key, a conflict would be caused (Heisenberg-style).
Thanks for clarifying!
Just to further clarify, the transactions (and therefore transaction conflict detection) are more flexible than simple CAS operations. If transaction A reads key K1, and then writes to key K2, it will be rejected as conflicting if transaction B writes to key K1 and is serialized as happening before transaction K1 is committed (even though no other transaction wrote to key K2).
Every transaction in FoundationDB is submitted as a collection of writes/mutations, but also contains records of all keys read (and the consistent version at which they were read). In this test, the transactions all have empty conflict ranges, and thus cannot conflict with other transactions (but we still have to check!).
DBs that are consistent until you observe them, a funny thought. In some parallel universe, the DB is still consistent; one could argue.
Heh, this is exactly why I could not understand what was being said.
[deleted]
all your wishes are granted: https://github.com/aphyr
To be sure, he revels in the discomfort of others too much. But he's a young kid just finding his voice, and his contributions to practical distributed database discussions are pretty much without parallel. Just ignore the bluster and you'll learn something new from kyle every day, even if it's about his buttcheeks.
Unfortunately this problem isn't specific to FoundationDB; the old "industry-standard" TPC-C benchmark has a similar low-contention design which has led to years of unrepresentative performance tuning and benchmarketing.
How is it unrepresentative? TPC-C involves several queries simulating new orders being entered, being updated, filled, paid, etc. It is very representative of the workload it is simulating. If you want to simulate a different workload, pick one of the other benchmarks instead of C.
Absolutely. Choosing the proper data model based on your access patterns is one of the best ways of keeping FDB performance high by reducing the likelihood of contention.
Yeah, this would have been more interesting if they could tell us what the throughput is given various levels of contention.