That's awesome. That's perfect tradeoff between the wishes of the parent website and convenience for your users.
I'm assuming you fetch feeds using a queue.
* How do you add jobs to the queue ? Do you run a cron every n minutes to check which feeds need fetching and add them to queue ?
* When you add jobs do you check to see if a previous job for fetching the queue is still in the queue or do you use flags to mark feeds for which the jobs are in queue and eliminate duplicate jobs that way ?
Yes, I use celery/rabbitmq as a queue. But I don't bother checking for duplicates. If the feed is still in the queue by the time it comes back around for processing again, I let it run. That means that the queue is behind enough that it will even out in the wash.
Comments
That's awesome. That's perfect tradeoff between the wishes of the parent website and convenience for your users.
I'm assuming you fetch feeds using a queue.
* How do you add jobs to the queue ? Do you run a cron every n minutes to check which feeds need fetching and add them to queue ?
* When you add jobs do you check to see if a previous job for fetching the queue is still in the queue or do you use flags to mark feeds for which the jobs are in queue and eliminate duplicate jobs that way ?
Yes, I use celery/rabbitmq as a queue. But I don't bother checking for duplicates. If the feed is still in the queue by the time it comes back around for processing again, I let it run. That means that the queue is behind enough that it will even out in the wash.