How do you deal with N-stage tasks in one of these message queues where you have to wind back the side-effects if something fails before the whole thing can be “done”?
I'm not the op, but in my little app that uses rabbitmq, you have to never assume timely delivery and processing of messages.
I use redis to keep a store of last updated timestamp property (that is, the last time the data was updated), and I essentially throw out a message if the timestamp is older.
For any writing changes I make I have a validation loop in the code path as a secondary validation.
I think once you get used to been defense about incoming messages you can handle most delays
Comments
How do you deal with N-stage tasks in one of these message queues where you have to wind back the side-effects if something fails before the whole thing can be “done”?
I'm not the op, but in my little app that uses rabbitmq, you have to never assume timely delivery and processing of messages.
I use redis to keep a store of last updated timestamp property (that is, the last time the data was updated), and I essentially throw out a message if the timestamp is older.
For any writing changes I make I have a validation loop in the code path as a secondary validation.
I think once you get used to been defense about incoming messages you can handle most delays
The saga pattern, or use something like Temporal.