Kafka is a lower-level system than RabbitMQ. It doesn't support:
* Topic routing. With RabbitMQ you can bind queue X to exchange Y with the routing keys "foo.bar." and "foo..baz". The queue will then get all messages matching those keys. That allows for very flexible pub-sub-style routing of messages; we use this extensively. As far as I can see, not really possible with Kafka without message duplication.
* Nacking model. Since Kafka queues are strictly linear, if a consumer fails to consume a message and wants to nack it and then process the remaining messages, it can't do so, since it has a single "read head". It would have to re-enqueue the message in that case.
* Prioritization. Not supported. You will have to create topics for different priorities and then consume those topics at a different rate depending on the priority.
* Message TTL.
And it's susceptible to network partitions, just like RabbitMQ.
I considered it, but it requires Apache ZooKeeper, which is yet another thing that needs to be configured and maintained. Not a fan of Java-based software either, to be honest.
Not a fan of Java-based software either, to be honest.
It makes me sad to see this is still something people say. I hope you reconsider this sentiment and investigate some of the really great Java-based software out there, especially Zookeeper and Kafka.
It's worth checking out how Zookeeper and Kafka did in aphyr's testing:
Every Java-based backed service I have come across, be it ElasticSearch, LogStash, Hadoop or PuppetDB, have all been memory-hogging beasts. Part of this is due to the GC, which tends to use more heap space than the program actually needs. Java is fast, but I have yet to see anyone claim it's lightweight.
Kafka seems to suffer from lack of partition tolerance, by the way, according to Aphyr. Not happy about the fact that it will just wipe a partition upon re-electing a new leader.
Comments
Apache Kafka seems like a promising alternative, although I haven't fully evaluated it yet.
Kafka is a lower-level system than RabbitMQ. It doesn't support:
* Topic routing. With RabbitMQ you can bind queue X to exchange Y with the routing keys "foo.bar." and "foo..baz". The queue will then get all messages matching those keys. That allows for very flexible pub-sub-style routing of messages; we use this extensively. As far as I can see, not really possible with Kafka without message duplication.
* Nacking model. Since Kafka queues are strictly linear, if a consumer fails to consume a message and wants to nack it and then process the remaining messages, it can't do so, since it has a single "read head". It would have to re-enqueue the message in that case.
* Prioritization. Not supported. You will have to create topics for different priorities and then consume those topics at a different rate depending on the priority.
* Message TTL.
And it's susceptible to network partitions, just like RabbitMQ.
I considered it, but it requires Apache ZooKeeper, which is yet another thing that needs to be configured and maintained. Not a fan of Java-based software either, to be honest.
It makes me sad to see this is still something people say. I hope you reconsider this sentiment and investigate some of the really great Java-based software out there, especially Zookeeper and Kafka.
It's worth checking out how Zookeeper and Kafka did in aphyr's testing:
http://aphyr.com/posts/291-call-me-maybe-zookeeper
http://aphyr.com/posts/293-call-me-maybe-kafka
Every Java-based backed service I have come across, be it ElasticSearch, LogStash, Hadoop or PuppetDB, have all been memory-hogging beasts. Part of this is due to the GC, which tends to use more heap space than the program actually needs. Java is fast, but I have yet to see anyone claim it's lightweight.
The worst part is "let's statically preallocate* the heap size like it's 1975".
*Note: never more than 31GB
Kafka seems to suffer from lack of partition tolerance, by the way, according to Aphyr. Not happy about the fact that it will just wipe a partition upon re-electing a new leader.
Yet so many of the services you use run on it.
Not sure that's an argument for anything. A lot of businesses build on crappy software.