RabbitMQ requiring a reliable network is causing problems for us in production. Anyone else struggled with this?
We're running several clusters on different providers; one is Digital Ocean, another is on a partner's VMware vMotion-based system. The kernel gets soft lockups now and then (in the case of vMotion, when VMs are automatically migrated to other physical nodes), which causes RabbitMQ partitions. The lockups may last a few seconds, but I've seen minute-long pauses.
When this happens, RabbitMQ starts throwing errors at clients. Queues disappear and clients can't do anything even though the local node is running. Although I understand the behaviour, that's not what I want from a queue; I want a local node to store messages until it rejoins the cluster and can dispatch them, and I want the local node to continue offering messages to those listening.
Unfortunately, RabbitMQ's design doesn't allow this: Queues live on the node they were created on, and RabbitMQ does not replicate them. We have turned on autohealing, but I don't like the fact that minority nodes simply wipe their data when they rejoin the cluster. The federation plugin doesn't look like a great solution.
I really like RabbitMQ, but maybe it's time to considering something else. Any suggestions? Something equally lightweight and "multimaster", but without the partition problem?
You need to look into using HA queues (ie. mirroring) and ensure you are using a client that properly supports consumer cancel notifications and are actually processing them correctly.
There is also significant TCP/IP and RabbitMQ tuning that can be done to make fail-over much faster etc.
I have used RabbitMQ for years and though it's not perfect if you do try something else you will quickly understand that it's still so far ahead of the competition.
This solution requires a RabbitMQ cluster, which means
that it will not cope seamlessly with network partitions
within the cluster and, for that reason, is not
recommended for use across a WAN
And then:
However, there is currently no way for a slave to know
whether or not its queue contents have diverged from the
master to which it is rejoining (this could happen
during a network partition, for example). As such, when
a slave rejoins a mirrored queue, it throws away any
durable local contents it already has and starts empty.
So I don't think that's helpful to us at all.
We are going to migrate to a client that supports consumer cancel notifications, though. Thanks for the tip.
True, what I do recommend though is using RMQ clusters on each cloud (where networking should be abit more reliable, the exception being AWS, which always sucks in this regard) then using federation (probably via shovel, but there are other means) to the other clouds.
Ultimately though.. when you get to this stage I question if your app is big enough to warrant this is suggest you use Azure Service Bus/Simple Queuing Services/whatever else your providers make available.
If your business really needs such control over messaging.. I understand. I have been in the situation where those easy ways out aren't available and I know your pain. Unfortunately there is no vendor you can go to make it go away, Tibco, Sterling etc aren't much better than RMQ.
I wish you the best of luck in your multi-cloud federated messaging system though, I highly suggest you look at Azure Service Bus though, I have nothing but praise for it despite being a devout RMQ zealot.
This will tune the TCP keepalives to decrease the time it takes for most client stacks to realize a server has gone away. It should also be configured on the servers themselves that are participating in the cluster.
As for rabbitmq tuning I recommend these settings at a minimum:
The Federation plugin looks interesting, but all the configuration that's necessary looks like a chore.
For one: Unlike most Unix daemons it must apparently be applied as rabbitmqctl commands after the daemon has started, which is awkward to do in a Puppet environment; as far as I can tell, you can get into situations where RabbitMQ starts in clustering mode, and there will be a race condition before you can apply the plugin config.
Secondly, the documentation is very poor. I have read everything pretty closely, but I still don't know if federation preserves the exact same behaviour, from the client's point of view, with regard to consistency, atomicity etc. What's missing is a feature matrix of how RabbitMQ behaves in the different modes.
The same thing goes for the Shovel plugin, except fortunately it allows for static config to be declared in rabbitmq.config. But again, because of the poor documentation, I still don't know how it would behave in practice.
Kafka is a lower-level system than RabbitMQ. It doesn't support:
* Topic routing. With RabbitMQ you can bind queue X to exchange Y with the routing keys "foo.bar." and "foo..baz". The queue will then get all messages matching those keys. That allows for very flexible pub-sub-style routing of messages; we use this extensively. As far as I can see, not really possible with Kafka without message duplication.
* Nacking model. Since Kafka queues are strictly linear, if a consumer fails to consume a message and wants to nack it and then process the remaining messages, it can't do so, since it has a single "read head". It would have to re-enqueue the message in that case.
* Prioritization. Not supported. You will have to create topics for different priorities and then consume those topics at a different rate depending on the priority.
* Message TTL.
And it's susceptible to network partitions, just like RabbitMQ.
I considered it, but it requires Apache ZooKeeper, which is yet another thing that needs to be configured and maintained. Not a fan of Java-based software either, to be honest.
Not a fan of Java-based software either, to be honest.
It makes me sad to see this is still something people say. I hope you reconsider this sentiment and investigate some of the really great Java-based software out there, especially Zookeeper and Kafka.
It's worth checking out how Zookeeper and Kafka did in aphyr's testing:
Every Java-based backed service I have come across, be it ElasticSearch, LogStash, Hadoop or PuppetDB, have all been memory-hogging beasts. Part of this is due to the GC, which tends to use more heap space than the program actually needs. Java is fast, but I have yet to see anyone claim it's lightweight.
Kafka seems to suffer from lack of partition tolerance, by the way, according to Aphyr. Not happy about the fact that it will just wipe a partition upon re-electing a new leader.
Comments
RabbitMQ requiring a reliable network is causing problems for us in production. Anyone else struggled with this?
We're running several clusters on different providers; one is Digital Ocean, another is on a partner's VMware vMotion-based system. The kernel gets soft lockups now and then (in the case of vMotion, when VMs are automatically migrated to other physical nodes), which causes RabbitMQ partitions. The lockups may last a few seconds, but I've seen minute-long pauses.
When this happens, RabbitMQ starts throwing errors at clients. Queues disappear and clients can't do anything even though the local node is running. Although I understand the behaviour, that's not what I want from a queue; I want a local node to store messages until it rejoins the cluster and can dispatch them, and I want the local node to continue offering messages to those listening.
Unfortunately, RabbitMQ's design doesn't allow this: Queues live on the node they were created on, and RabbitMQ does not replicate them. We have turned on autohealing, but I don't like the fact that minority nodes simply wipe their data when they rejoin the cluster. The federation plugin doesn't look like a great solution.
I really like RabbitMQ, but maybe it's time to considering something else. Any suggestions? Something equally lightweight and "multimaster", but without the partition problem?
You need to look into using HA queues (ie. mirroring) and ensure you are using a client that properly supports consumer cancel notifications and are actually processing them correctly.
There is also significant TCP/IP and RabbitMQ tuning that can be done to make fail-over much faster etc.
I have used RabbitMQ for years and though it's not perfect if you do try something else you will quickly understand that it's still so far ahead of the competition.
Best of luck!
HA queues have this caveat in the documentation:
And then: So I don't think that's helpful to us at all.We are going to migrate to a client that supports consumer cancel notifications, though. Thanks for the tip.
True, what I do recommend though is using RMQ clusters on each cloud (where networking should be abit more reliable, the exception being AWS, which always sucks in this regard) then using federation (probably via shovel, but there are other means) to the other clouds.
Ultimately though.. when you get to this stage I question if your app is big enough to warrant this is suggest you use Azure Service Bus/Simple Queuing Services/whatever else your providers make available.
If your business really needs such control over messaging.. I understand. I have been in the situation where those easy ways out aren't available and I know your pain. Unfortunately there is no vendor you can go to make it go away, Tibco, Sterling etc aren't much better than RMQ.
I wish you the best of luck in your multi-cloud federated messaging system though, I highly suggest you look at Azure Service Bus though, I have nothing but praise for it despite being a devout RMQ zealot.
You misunderstood me, I think. Our clouds aren't connected. The partitioning problem exists within each data center (eg., Digital Ocean).
So federation/shovel is probably not the solution.
SQS is far too simple for our needs. No idea what Azure is, but if it's SaaS the latency will likely be too high. We need local performance.
Can you point me at a good place to start for RMQ and TCP/IP stack tuning?
Sure, I would recommend the below in sysctl:
This will tune the TCP keepalives to decrease the time it takes for most client stacks to realize a server has gone away. It should also be configured on the servers themselves that are participating in the cluster.As for rabbitmq tuning I recommend these settings at a minimum:
Thank you! I have some reading to do. Appreciate it!
Presumably running clustered configuration on a public VM hosting provider is not a recommended configuration.
Kernel lockups lasting minutes at a time seems like a serious issue a lot of software might have problems with.
Have you tried federation of shovel modes?
The Federation plugin looks interesting, but all the configuration that's necessary looks like a chore.
For one: Unlike most Unix daemons it must apparently be applied as rabbitmqctl commands after the daemon has started, which is awkward to do in a Puppet environment; as far as I can tell, you can get into situations where RabbitMQ starts in clustering mode, and there will be a race condition before you can apply the plugin config.
Secondly, the documentation is very poor. I have read everything pretty closely, but I still don't know if federation preserves the exact same behaviour, from the client's point of view, with regard to consistency, atomicity etc. What's missing is a feature matrix of how RabbitMQ behaves in the different modes.
The same thing goes for the Shovel plugin, except fortunately it allows for static config to be declared in rabbitmq.config. But again, because of the poor documentation, I still don't know how it would behave in practice.
Apache Kafka seems like a promising alternative, although I haven't fully evaluated it yet.
Kafka is a lower-level system than RabbitMQ. It doesn't support:
* Topic routing. With RabbitMQ you can bind queue X to exchange Y with the routing keys "foo.bar." and "foo..baz". The queue will then get all messages matching those keys. That allows for very flexible pub-sub-style routing of messages; we use this extensively. As far as I can see, not really possible with Kafka without message duplication.
* Nacking model. Since Kafka queues are strictly linear, if a consumer fails to consume a message and wants to nack it and then process the remaining messages, it can't do so, since it has a single "read head". It would have to re-enqueue the message in that case.
* Prioritization. Not supported. You will have to create topics for different priorities and then consume those topics at a different rate depending on the priority.
* Message TTL.
And it's susceptible to network partitions, just like RabbitMQ.
I considered it, but it requires Apache ZooKeeper, which is yet another thing that needs to be configured and maintained. Not a fan of Java-based software either, to be honest.
It makes me sad to see this is still something people say. I hope you reconsider this sentiment and investigate some of the really great Java-based software out there, especially Zookeeper and Kafka.
It's worth checking out how Zookeeper and Kafka did in aphyr's testing:
http://aphyr.com/posts/291-call-me-maybe-zookeeper
http://aphyr.com/posts/293-call-me-maybe-kafka
Every Java-based backed service I have come across, be it ElasticSearch, LogStash, Hadoop or PuppetDB, have all been memory-hogging beasts. Part of this is due to the GC, which tends to use more heap space than the program actually needs. Java is fast, but I have yet to see anyone claim it's lightweight.
The worst part is "let's statically preallocate* the heap size like it's 1975".
*Note: never more than 31GB
Kafka seems to suffer from lack of partition tolerance, by the way, according to Aphyr. Not happy about the fact that it will just wipe a partition upon re-electing a new leader.
Yet so many of the services you use run on it.
Not sure that's an argument for anything. A lot of businesses build on crappy software.