The "queue" is merely a Redis list. Durable by default.
Redis PUB/SUB is not suited for the task queues we use, because any number of subscribers will receive the messages. We want to guarantee that only one worker will act upon each message.
Err, I guess I meant something else. For instance, what happens if one of the workers goes down after popping? The message is lost, right?
I think I invented your architecture in my head that isn't near reality. I imagined you were using the workers to take incoming published messages and pushing them into the queues that subscribed connections are popping off of. Effectively building your own fanout. So in this context, I was wondering why not just use pubsub which will handle fanout and get rid of the entire worker model.
Comments
Nice post.
Do you do anything special to make the queue which your workers are BLPOPing durable?
Is there a reason you didn't use Redis pub/sub? Seems like the perfect use case.
The "queue" is merely a Redis list. Durable by default.
Redis PUB/SUB is not suited for the task queues we use, because any number of subscribers will receive the messages. We want to guarantee that only one worker will act upon each message.
Err, I guess I meant something else. For instance, what happens if one of the workers goes down after popping? The message is lost, right?
I think I invented your architecture in my head that isn't near reality. I imagined you were using the workers to take incoming published messages and pushing them into the queues that subscribed connections are popping off of. Effectively building your own fanout. So in this context, I was wondering why not just use pubsub which will handle fanout and get rid of the entire worker model.
Thanks for the reply!