It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC".
Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages.
So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).
I'm not familiar at all with RabbitMQ, so can't really comment, but I'm pretty sure they give guarantees like a producer can wait until a given message is consumed. This means there's no fire-and-forget, even though the message is logged to disk at one point, you need to do all that per message book keeping.
It is more accurate to say that RabbitMQ supports both fire-and-forget and producer-waits. The exact behavior is specified by a combination of how you configure exchanges and queues, and per-message settings, and how you write your client code. For example, your application can decide that some of the messages it injects into a queue are to be durable and others not. It is quite flexible (though the docs are lacking when it comes to specific advice for various use cases).
Comments
It's an architecture thing. Most message queues are written the way you'd initially think to manage a message queue, you keep a big queue of objects in memory, and in order to get delivery guarantees, you have to hold on to them until the consumer confirms receipt. This leads to pathological garbage collection scenarios, the old "holding onto an object just long enough to make it really expensive to GC".
Kafka, on the other hand, when you write a message to the broker the broker writes it immediately to disk queue rather than holding it in memory. But isn't that slower? No, it's not, because it's in page cache, which is managed more efficiently than garbage collected memory. Then, when consuming, rather than keeping metrics for each individual message being received, consumers simply have a log position -- they periodically commit, which tells the broker that all of the messages until that point have been consumed. If they never commit, eventually another consumer will get those messages.
So basically, it scales a ton better because you're just doing scads of sequential I/O with occasional commits, rather than tracking a bunch of messages in memory individually (which in theory should be fast but causes GC problems).
How does that compare to RabbitMQ's disk backed store?
I'm not familiar at all with RabbitMQ, so can't really comment, but I'm pretty sure they give guarantees like a producer can wait until a given message is consumed. This means there's no fire-and-forget, even though the message is logged to disk at one point, you need to do all that per message book keeping.
It is more accurate to say that RabbitMQ supports both fire-and-forget and producer-waits. The exact behavior is specified by a combination of how you configure exchanges and queues, and per-message settings, and how you write your client code. For example, your application can decide that some of the messages it injects into a queue are to be durable and others not. It is quite flexible (though the docs are lacking when it comes to specific advice for various use cases).
How does it compare to services like Amazon SNS, IronIO, 'etc in terms of costs and other tradeoffs?
Since you host it yourself you can't compare it because you on the variable cost of servers and you time to maintain them.