My usual solution is to add checks in the cron job to make sure they don't repeat or duplicate anything, by using an audit table. So for example when a celery tasks triggers an email, I store an event called EMAIL_X_SENT to the audit table with meta data and check it later before sending it again.
Of course it complicates the logic a bit but I've noticed it's the same pattern that works everywhere so I just abstracted most of into a custom task class.
Another way typically is to use a shared lock just like above except in the cache backend. So you could probably extend highlander to use cache backend etc.
Comments
Does this work with celery tasks?
My usual solution is to add checks in the cron job to make sure they don't repeat or duplicate anything, by using an audit table. So for example when a celery tasks triggers an email, I store an event called EMAIL_X_SENT to the audit table with meta data and check it later before sending it again.
Of course it complicates the logic a bit but I've noticed it's the same pattern that works everywhere so I just abstracted most of into a custom task class.
Another way typically is to use a shared lock just like above except in the cache backend. So you could probably extend highlander to use cache backend etc.
I think you could use this with celery tasks so as long as each worker used a different PID file.