With a lot of effort by many folks in the community, HBase has mostly tackled the full-GC problem, but still has occasional issues with some workloads.
So, GC was definitely one factor - not having GC means we can give 99th percentile numbers in the single-digit milliseconds, which is pretty nice. Our master process actually has shown <1ms 99.99th percentile for tablet location requests on an 80 node cluster. So again, numbers that are super difficult to get on the JVM unless you take an allocation-free approach like the HFT guys do.
Another factor was ease of integration of platform-specific code for performance reasons. For example, we make use of SSE prefetch instructions to improve scan speed in our concurrent B-tree by 30% or so. The b-tree itself would be difficult to implement in Java due to lack of control over object layout, etc. While you can eventually get the same performance with enough off-heaping and sun.misc.Unsafe, my feeling is that, by the time you've gone down that road, you might as well be using C++.
I'll admit that, after many years of not writing native code, I was a bit nervous of diving back in. Segfaults are never fun. But, we soon realized that the native code tooling has improved a _ton_ in the last decade. We run all of our tests precommit using the excellent Sanitizer tools from Google (ThreadSanitizer, AddressSanitizer, LeakSanitizer) and those make it nearly trivial to diagnose a leak or crash. We also have pretty strict guidelines around use of pointers, based on the Google C++ guidelines. Many will complain that this is a neutered form of C++, and they're right. But it's also a relatively safe form of C++.
I could probably write a lengthy blog post on our experiences of C++ vs Java, but hopefully the above gives you a taste. Overall I've been happy with the decision. Slightly more time spent on crashes. Less time spent on chasing hard-to-reproduce performance or memory consumption issues. And the thread checking tools are actually far superior, so I'd say less time spent chasing races.
+1 on writing up a blog post on c++ vs java. I suspect given your background in java, people may heed your words a bit more than usual. There's definitely a lot of outdated thinking in java land with regards to (modern) c++ and accompanying toolchain. Many big data projects could benefit from being written in c++ rather than java (or another jvm language).
Thanks for the response, Todd!
Since most projects in Hadoop ecosystem are written in Java/Scala on JVM I was pleasantly surprised about choice of C++ :)
Comments
I spent a lot of time in 2011 or so struggling with GC on the JVM: http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-h... has some of the gory details. Even hacked a bit on G1: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2011-A...
With a lot of effort by many folks in the community, HBase has mostly tackled the full-GC problem, but still has occasional issues with some workloads.
So, GC was definitely one factor - not having GC means we can give 99th percentile numbers in the single-digit milliseconds, which is pretty nice. Our master process actually has shown <1ms 99.99th percentile for tablet location requests on an 80 node cluster. So again, numbers that are super difficult to get on the JVM unless you take an allocation-free approach like the HFT guys do.
Another factor was ease of integration of platform-specific code for performance reasons. For example, we make use of SSE prefetch instructions to improve scan speed in our concurrent B-tree by 30% or so. The b-tree itself would be difficult to implement in Java due to lack of control over object layout, etc. While you can eventually get the same performance with enough off-heaping and sun.misc.Unsafe, my feeling is that, by the time you've gone down that road, you might as well be using C++.
I'll admit that, after many years of not writing native code, I was a bit nervous of diving back in. Segfaults are never fun. But, we soon realized that the native code tooling has improved a _ton_ in the last decade. We run all of our tests precommit using the excellent Sanitizer tools from Google (ThreadSanitizer, AddressSanitizer, LeakSanitizer) and those make it nearly trivial to diagnose a leak or crash. We also have pretty strict guidelines around use of pointers, based on the Google C++ guidelines. Many will complain that this is a neutered form of C++, and they're right. But it's also a relatively safe form of C++.
I could probably write a lengthy blog post on our experiences of C++ vs Java, but hopefully the above gives you a taste. Overall I've been happy with the decision. Slightly more time spent on crashes. Less time spent on chasing hard-to-reproduce performance or memory consumption issues. And the thread checking tools are actually far superior, so I'd say less time spent chasing races.
I think writing a long post about your experiences of C++ vs Java would be great (I'd pay you in [choose your drinks and count] for it ;)).
Quick example of how TSAN makes it easy to understand and fix races -- here's a commit message which shows its output: https://github.com/cloudera/kudu/commit/e402d5ed79a9c98283b6...
Apparently Google has this same tool for Java internally, but hasn't open sourced it due to their litigation with Oracle over Java stuff.
I'd be interested in reading that. There's still a lot of FUD around using C++ for new projects.
+1 on writing up a blog post on c++ vs java. I suspect given your background in java, people may heed your words a bit more than usual. There's definitely a lot of outdated thinking in java land with regards to (modern) c++ and accompanying toolchain. Many big data projects could benefit from being written in c++ rather than java (or another jvm language).
Thanks for the response, Todd! Since most projects in Hadoop ecosystem are written in Java/Scala on JVM I was pleasantly surprised about choice of C++ :)