the author has corrected the discrepancies since then
As of the time I posted this comment, the blog post still seems to be comparing indexed MongoDB operations against non-indexed RethinkDB operations. Under those conditions I'd expect RethinkDB to be at least 1000x slower than MongoDB. The fact that he's finding that RethinkDB is only 3x slower than MongoDB makes me think that there are still other major problems with this benchmark.
No, but at least he tried...
It's true that the author tried; but that doesn't change the fact that people are going to read this blog post and assume that the numbers are at least approximately correct. As a RethinkDB employee, it really frustrates me to see RethinkDB being judged according to benchmarks that are conducted so carelessly that they are essentially random.
I think this is the fourth time in the past year that I've seen a third party try to benchmark RethinkDB and get something wrong. Maybe we need to start a "best practice" of checking in with the maintainers of a project before publishing benchmark results about the project.
Some mistakes in his benchmarks, among probably others:
- I don't see any mongodb index creation, so mongodb is inserting with no index while rethinkdb is inserting with the index. That's probably why there's a gap between the two
- there's no mongodb index, and rethinkdb queries do not make use of the index (this is probably why rethinkdb is not 1000x slower: both aren't using indexes)
- the $in query should be last_update: random_timestamp(), there's no need for $in here
- his insertion code creates 100K memory clones of the object to insert in the mongodb version only, not in rethinkdb
I'm sad to add: what the author is benchmarking here is the likely performance of a system he could build with either db. It's not necessarily bad (save for bad press) that he's bad at benchmarking: the mistakes he's made in his benchmark are similar to the mistakes he'll make in his code.
In the benchmark script [1] that the author provided on his GitHub account, there's a call to ensure_index() for MongoDB. And he's reporting an average latency of 0.15ms for MongoDB read operations, so it's pretty clear that the MongoDB index is actually being used.
Yes, this kind of latency definitely says "indexed".
When I wrote this, I'd read a rethinkDB employee say "it's not using the index on rethinkdb", and performance was similar (3x) between the two. I trusted the "no index" path, and couldn't find an ensureIndex command... So I assumed there was no index on mongodb.
Truth is, this kind of performance can only come with indexes, on RethinkDB and mongodb.
RethinkDB needs to ship its own benchmark client. Also implement an ugh YCSB driver. Provide both with the database download.
It's madness to expect someone new to database benchmarking to implement a correct fully featured benchmark client. They are going to stumble enough on database and instance configuration as it is.
Not sure why you are frustrated it's just a blog post by someone who was inexperienced with your product. At least he owned up to the mistakes and was willing to fix it. It's an opportunity for you to work with the guy to show him how to do it properly and write a blog post of your own.
I would say that you probably should look at your API because I've never used a database that required me to explicitly define which indexes I want to use for a read. But I've never used RethinkDB so maybe there is a legitimate reason.
GP is frustrated because it's not just the one blog post. Even if the author is willing to correct it, the original bad data will tend to get more exposure, because most people won't check in for corrections (unless they see a post like OP). Then, there'll be another crappy benchmark next month or next week. If I thought my product was being judged this way, it would drive me frothing mad.
Comments
As of the time I posted this comment, the blog post still seems to be comparing indexed MongoDB operations against non-indexed RethinkDB operations. Under those conditions I'd expect RethinkDB to be at least 1000x slower than MongoDB. The fact that he's finding that RethinkDB is only 3x slower than MongoDB makes me think that there are still other major problems with this benchmark.
It's true that the author tried; but that doesn't change the fact that people are going to read this blog post and assume that the numbers are at least approximately correct. As a RethinkDB employee, it really frustrates me to see RethinkDB being judged according to benchmarks that are conducted so carelessly that they are essentially random.
I think this is the fourth time in the past year that I've seen a third party try to benchmark RethinkDB and get something wrong. Maybe we need to start a "best practice" of checking in with the maintainers of a project before publishing benchmark results about the project.
Some mistakes in his benchmarks, among probably others:
- I don't see any mongodb index creation, so mongodb is inserting with no index while rethinkdb is inserting with the index. That's probably why there's a gap between the two
- there's no mongodb index, and rethinkdb queries do not make use of the index (this is probably why rethinkdb is not 1000x slower: both aren't using indexes)
- the $in query should be last_update: random_timestamp(), there's no need for $in here
- his insertion code creates 100K memory clones of the object to insert in the mongodb version only, not in rethinkdb
I'm sad to add: what the author is benchmarking here is the likely performance of a system he could build with either db. It's not necessarily bad (save for bad press) that he's bad at benchmarking: the mistakes he's made in his benchmark are similar to the mistakes he'll make in his code.
But yes, the author may use some help!
In the benchmark script [1] that the author provided on his GitHub account, there's a call to ensure_index() for MongoDB. And he's reporting an average latency of 0.15ms for MongoDB read operations, so it's pretty clear that the MongoDB index is actually being used.
[1] https://github.com/martinrusev/rethinkvsmongo-benchmark/blob...
Yes, this kind of latency definitely says "indexed".
When I wrote this, I'd read a rethinkDB employee say "it's not using the index on rethinkdb", and performance was similar (3x) between the two. I trusted the "no index" path, and couldn't find an ensureIndex command... So I assumed there was no index on mongodb.
Truth is, this kind of performance can only come with indexes, on RethinkDB and mongodb.
RethinkDB needs to ship its own benchmark client. Also implement an ugh YCSB driver. Provide both with the database download.
It's madness to expect someone new to database benchmarking to implement a correct fully featured benchmark client. They are going to stumble enough on database and instance configuration as it is.
http://en.wikipedia.org/wiki/David_DeWitt
@threeseed
In a way, it's traditional (IBM IMS/DB, 1960s).
+1 on 'Profiling best practice'. Any such existing project?
Not sure why you are frustrated it's just a blog post by someone who was inexperienced with your product. At least he owned up to the mistakes and was willing to fix it. It's an opportunity for you to work with the guy to show him how to do it properly and write a blog post of your own.
I would say that you probably should look at your API because I've never used a database that required me to explicitly define which indexes I want to use for a read. But I've never used RethinkDB so maybe there is a legitimate reason.
GP is frustrated because it's not just the one blog post. Even if the author is willing to correct it, the original bad data will tend to get more exposure, because most people won't check in for corrections (unless they see a post like OP). Then, there'll be another crappy benchmark next month or next week. If I thought my product was being judged this way, it would drive me frothing mad.