In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ.
The actual cases where messages are handled multiple times is limited to when a client disappeared during message processing (a hard restart) or it passed the allowable time window to respond and was given to another client.
If there are any specific numbers you are curious about, please ask.
When people say "billions of messages per day" do they realize that 1 billion messages =~ 11500 msgs/sec ? RabbitMQ as well as any self respecting message broker is able to handle that on a single machine
Comments
Some of the background behind our decisions are on our blog post http://word.bitly.com/post/33232969144/nsq
In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ.
The actual cases where messages are handled multiple times is limited to when a client disappeared during message processing (a hard restart) or it passed the allowable time window to respond and was given to another client.
If there are any specific numbers you are curious about, please ask.
When people say "billions of messages per day" do they realize that 1 billion messages =~ 11500 msgs/sec ? RabbitMQ as well as any self respecting message broker is able to handle that on a single machine