I always encounter the problem where I write Python scripts that run on a cron job that sometimes take longer than the interval before the same cron job will run again (e.g., I have a cron that runs every hour and one run takes 2 hours to complete). In this scenario, you would want the first cron to complete before the second cron is run. What Highlander does is if it sees that your cron is already running, it immediately returns thereby skipping that cron run.
My usual solution is to add checks in the cron job to make sure they don't repeat or duplicate anything, by using an audit table. So for example when a celery tasks triggers an email, I store an event called EMAIL_X_SENT to the audit table with meta data and check it later before sending it again.
Of course it complicates the logic a bit but I've noticed it's the same pattern that works everywhere so I just abstracted most of into a custom task class.
Another way typically is to use a shared lock just like above except in the cache backend. So you could probably extend highlander to use cache backend etc.
Easy enough to just use halockrun from the hatools package for that. That is also its raison d'être really. Sure it is c but it is pretty much done as a tool over fcntl(3).
Comments
I always encounter the problem where I write Python scripts that run on a cron job that sometimes take longer than the interval before the same cron job will run again (e.g., I have a cron that runs every hour and one run takes 2 hours to complete). In this scenario, you would want the first cron to complete before the second cron is run. What Highlander does is if it sees that your cron is already running, it immediately returns thereby skipping that cron run.
Does this work with celery tasks?
My usual solution is to add checks in the cron job to make sure they don't repeat or duplicate anything, by using an audit table. So for example when a celery tasks triggers an email, I store an event called EMAIL_X_SENT to the audit table with meta data and check it later before sending it again.
Of course it complicates the logic a bit but I've noticed it's the same pattern that works everywhere so I just abstracted most of into a custom task class.
Another way typically is to use a shared lock just like above except in the cache backend. So you could probably extend highlander to use cache backend etc.
I think you could use this with celery tasks so as long as each worker used a different PID file.
By the time I reach this level of complexity, I usually think that what I really want is a daemon.
But I suppose this is a good way to split the difference.
Easy enough to just use halockrun from the hatools package for that. That is also its raison d'être really. Sure it is c but it is pretty much done as a tool over fcntl(3).
http://www.fatalmind.com/software/hatools/
What would be the difference between this and using something like APScheduler? Doesn't it achieve the same thing?
This is a much simpler solution to a less complex problem.
So do you not write your scripts to know if its already running using file locks?
I do, but now with Highlander I've created a generic solution.