This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost.
I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies.
One solution is to stand up redundant nsqd pairs (on separate hosts) that receive copies of the same portion of messages.
OK, delivery is only guaranteed if I run multiple independent sets of NSQd and write messages to both.
Sequence numbers on messages sent by producers coupled with some sort of persistence by the sender. This way when a gap is detected some sort of resend protocol can be implemented to achieve the delivery guarantee.
Comments
This ensures that the only edge case that would result in message loss is an unclean shutdown of an nsqd process. In that case, any messages that were in memory (or any buffered writes not flushed to disk) would be lost.
I don't understand how you can call it a message delivery "guarantee" when you're susceptible to losing messages when a node dies.
One solution is to stand up redundant nsqd pairs (on separate hosts) that receive copies of the same portion of messages.
OK, delivery is only guaranteed if I run multiple independent sets of NSQd and write messages to both.
Regardless, it looks like an interesting project.
How would you design a system that didn't have this problem at the individual node level?
A similar design to TCP/IP - sequence numbers, acknowledgements & resending capability on sender side
Sequence numbers on messages sent by producers coupled with some sort of persistence by the sender. This way when a gap is detected some sort of resend protocol can be implemented to achieve the delivery guarantee.
So the sender just keeps trying until it gets an ACK. Fair enough, though I imagine you'd want to run multiple nodes even with this.
Correct. Reliability and Availability are two separate but related goals.
This is a solved problem in most messaging protocols.