If you're building a distributed KV stores, you should benchmark against other distributed KV stores. Mongo and Cassandra aren't really. But Riak is.
Plus, since its "distributed" here's the benchmark I'd like to see:
1. Set up a cluster of 8 nodes. Set data replication to 3.
2. Load 3TB of data into the cluster, across 1M documents (or some data set of that order)
3. Run your tests. Optimize each of the DBs for the best way to access them (Eg: link walking vs. map reduce on Riak if that's faster, or secondary indexes if that's faster, or Riak Search if that's faster-- there are many ways to search Riak.)
4. Throw out the results of #3.
5. Pull the plug on 2 nodes. EG: Shut down completely, no longer on the net work at all. Poof, gone. Pick the two nodes by rolling the dice. If you have SPFs, and a bad role of the dice would have brought the whole cluster down, remove the word "distributed" from your marketing.
6. Let the cluster sit for 30 minutes.
7. Run your benchmarks.
The benchmarks developed in #7 are the ones I want to see.
I have no clue how HyperDex would perform in this situation. It could kick Riak's butt (but then, small cluster performance is just one of the criteria that is important to me). I just wish people did benchmarks like this (though I know its a PITA to do it this way.)
Why would a competitor build this Riak benchmark? How are they supposed to know which Riak configuration performs best?
That's the point of YCSB. Each vendor can submit the optimal configuration for their system, and they all run the same benchmark.
At the end of the day, each vendor is going to publish benchmarks that show their system performing better than all others.
It's your job, not theirs, to verify those benchmarks for your particular work load.
Edit: Also, while Mongo wouldn't classify as a distributed store, Cassandra definitely would. In fact, it's more "distributed" than Riak, since you have to pay Basho for multiple DC support (unless that's in the open source version?), whereas Cassandra is completely free.
Have you looked at YCSB? It's a pathologically poorly designed benchmark. It essentially tests how good a developer is at implementing a Java wrapper around a toy object model and interfacing that with a database.
A proper benchmark would hold constant things which are reasonably expected to be constant: the use case scenario, the data, the warmup requirements, the concurrency, and the count. If feeling frisky, the hardware, operating system and networking environment. And then step back.
For example, "On an EC2 Extra Large instance running Ubuntu 11.10 with whatever tuning the package recommends, what is the transactions per second when 50 vendor-provided clients on the same LAN are attempting to write and read random records out of a pool of 10mm 5k records?"
In my opinion they're both KV stores. Where they differ is what data structure they store. Cassandra stores BigTable rows indexed by a key and Riak is a binary blob + metadata indexed by a key.
I suppose it all depends on how be define a key/value store. Is it simply a data structure that can be accessed by a primary key. If that's the fact, nearly all databases fall under that description. If, on the other hand, a key/value store is simply a database that stores a blob indexed by a key, then nirvana is correct in saying "Cassandra" is not a KV store, but where does that get us?
I think that comparing HyperDex to Cassandra is a valid comparison. Both support pk lookups and secondary key lookups. The same is true about Riak. It would be nice to see a YCSB benchmark for properly configured Riak cluster.
I am pretty ignorant of the YCSB benchmark but it would be nice to have a benchmark that tests fault tolerance as well as raw performance.
Comments
If you're building a distributed KV stores, you should benchmark against other distributed KV stores. Mongo and Cassandra aren't really. But Riak is.
Plus, since its "distributed" here's the benchmark I'd like to see:
1. Set up a cluster of 8 nodes. Set data replication to 3. 2. Load 3TB of data into the cluster, across 1M documents (or some data set of that order) 3. Run your tests. Optimize each of the DBs for the best way to access them (Eg: link walking vs. map reduce on Riak if that's faster, or secondary indexes if that's faster, or Riak Search if that's faster-- there are many ways to search Riak.) 4. Throw out the results of #3. 5. Pull the plug on 2 nodes. EG: Shut down completely, no longer on the net work at all. Poof, gone. Pick the two nodes by rolling the dice. If you have SPFs, and a bad role of the dice would have brought the whole cluster down, remove the word "distributed" from your marketing. 6. Let the cluster sit for 30 minutes. 7. Run your benchmarks.
The benchmarks developed in #7 are the ones I want to see.
I have no clue how HyperDex would perform in this situation. It could kick Riak's butt (but then, small cluster performance is just one of the criteria that is important to me). I just wish people did benchmarks like this (though I know its a PITA to do it this way.)
Why would a competitor build this Riak benchmark? How are they supposed to know which Riak configuration performs best?
That's the point of YCSB. Each vendor can submit the optimal configuration for their system, and they all run the same benchmark.
At the end of the day, each vendor is going to publish benchmarks that show their system performing better than all others.
It's your job, not theirs, to verify those benchmarks for your particular work load.
Edit: Also, while Mongo wouldn't classify as a distributed store, Cassandra definitely would. In fact, it's more "distributed" than Riak, since you have to pay Basho for multiple DC support (unless that's in the open source version?), whereas Cassandra is completely free.
If things haven't changed in the past year you are right about not being able to do replicated multiple DC support.
Have you looked at YCSB? It's a pathologically poorly designed benchmark. It essentially tests how good a developer is at implementing a Java wrapper around a toy object model and interfacing that with a database.
A proper benchmark would hold constant things which are reasonably expected to be constant: the use case scenario, the data, the warmup requirements, the concurrency, and the count. If feeling frisky, the hardware, operating system and networking environment. And then step back.
For example, "On an EC2 Extra Large instance running Ubuntu 11.10 with whatever tuning the package recommends, what is the transactions per second when 50 vendor-provided clients on the same LAN are attempting to write and read random records out of a pool of 10mm 5k records?"
What exactly makes Cassandra "not really" a KV store, while Riak "really" is a KV store? I'm calling No True Scotsman.
In my opinion they're both KV stores. Where they differ is what data structure they store. Cassandra stores BigTable rows indexed by a key and Riak is a binary blob + metadata indexed by a key.
I suppose it all depends on how be define a key/value store. Is it simply a data structure that can be accessed by a primary key. If that's the fact, nearly all databases fall under that description. If, on the other hand, a key/value store is simply a database that stores a blob indexed by a key, then nirvana is correct in saying "Cassandra" is not a KV store, but where does that get us?
I think that comparing HyperDex to Cassandra is a valid comparison. Both support pk lookups and secondary key lookups. The same is true about Riak. It would be nice to see a YCSB benchmark for properly configured Riak cluster.
I am pretty ignorant of the YCSB benchmark but it would be nice to have a benchmark that tests fault tolerance as well as raw performance.