Yes, I use celery/rabbitmq as a queue. But I don't bother checking for duplicates. If the feed is still in the queue by the time it comes back around for processing again, I let it run. That means that the queue is behind enough that it will even out in the wash.