Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and why one might need Kafka?
It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC".
Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages.
So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).
I'm not familiar at all with RabbitMQ, so can't really comment, but I'm pretty sure they give guarantees like a producer can wait until a given message is consumed. This means there's no fire-and-forget, even though the message is logged to disk at one point, you need to do all that per message book keeping.
It is more accurate to say that RabbitMQ supports both fire-and-forget and producer-waits. The exact behavior is specified by a combination of how you configure exchanges and queues, and per-message settings, and how you write your client code. For example, your application can decide that some of the messages it injects into a queue are to be durable and others not. It is quite flexible (though the docs are lacking when it comes to specific advice for various use cases).
It trades reliability for throughput. So event tracking logs in realtome is the most common use case I think, where there's a ton of data. But we have a mapreduce calculate at the end of the day from raw logs for accuracy.
Which part is not correct? They go into great detail about their design for throughput. Re: reliability, "Not all use cases require such strong guarantees. For uses which are latency sensitive we allow the producer to specify the durability level it desires."
Pre-0.8, if a machine fails you lose all the data on that machine, only the lower durability levels were available. It guarantees at least once processing, while other queues generally make stronger claims. etc.
Before 0.8, you are correct that if you lose a disk, you lose the data. However, just because a broker goes down, that doesn't mean you lose the data on it - it just becomes unavailable to consumers. The log files backing Kafka partitions are fully durable append only files. So the durability guarantees are pretty good, even before 0.8.
What you're talking about is failover and fault tolerance, which are greatly improved in 0.8 with the addition of replication.
Comments
Have read a bit of the intro material, but I'm still not grokking what makes Kafka fundamentally different from ActiveMQ / Apollo. Can anyone sum up where and why one might need Kafka?
It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC".
Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages.
So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).
How does that compare to RabbitMQ's disk backed store?
I'm not familiar at all with RabbitMQ, so can't really comment, but I'm pretty sure they give guarantees like a producer can wait until a given message is consumed. This means there's no fire-and-forget, even though the message is logged to disk at one point, you need to do all that per message book keeping.
It is more accurate to say that RabbitMQ supports both fire-and-forget and producer-waits. The exact behavior is specified by a combination of how you configure exchanges and queues, and per-message settings, and how you write your client code. For example, your application can decide that some of the messages it injects into a queue are to be durable and others not. It is quite flexible (though the docs are lacking when it comes to specific advice for various use cases).
How does it compare to services like Amazon SNS, IronIO, 'etc in terms of costs and other tradeoffs?
Since you host it yourself you can't compare it because you on the variable cost of servers and you time to maintain them.
In Kafka, topics are (partitioned) streams of messages. Consumers of a topic keep track of their "cursor" in that stream.
EDIT: should add that morkbot had a great link too:
https://news.ycombinator.com/item?id=6874607 http://www.quora.com/RabbitMQ/RabbitMQ-vs-Kafka-which-one-fo...
Stores messages for default 2 days, and messages can be replayed. Plus distributed servers.
It trades reliability for throughput. So event tracking logs in realtome is the most common use case I think, where there's a ton of data. But we have a mapreduce calculate at the end of the day from raw logs for accuracy.
This is not correct: http://kafka.apache.org/documentation.html#design
:-)
Which part is not correct? They go into great detail about their design for throughput. Re: reliability, "Not all use cases require such strong guarantees. For uses which are latency sensitive we allow the producer to specify the durability level it desires."
Pre-0.8, if a machine fails you lose all the data on that machine, only the lower durability levels were available. It guarantees at least once processing, while other queues generally make stronger claims. etc.
It's still very good at what it does.
Before 0.8, you are correct that if you lose a disk, you lose the data. However, just because a broker goes down, that doesn't mean you lose the data on it - it just becomes unavailable to consumers. The log files backing Kafka partitions are fully durable append only files. So the durability guarantees are pretty good, even before 0.8.
What you're talking about is failover and fault tolerance, which are greatly improved in 0.8 with the addition of replication.
Use cases (from the project page): http://kafka.apache.org/documentation.html#uses