As someone who has had to write this themselves multiple times, there are a few bits that I consider to be missing:
1) Command line verification - is the pid owned by the same type of process as is running now? PIDs are re-used, ensure it's the same (the creation time check helps, but it doesn't say anything about what process wrote it).
2) Process Hang Detection - Has the process actually consumed any CPU ticks in the last minute?
3) Infinite loop detection - Is the other process stuck processing something uselessly?
4) Killing off stuck processes - 2 or 3 true? Behead it and continue on. Optionally do some form of alerting - stderr is probably fine.
Add these, and I would personally find it much more useful.
1. I assume that it's the same type of process because by default the PID file is being written to the current working directory of the script. If you'd like, you can specify a location yourself to ensure that each type of process is grouped on one PID file.
2. Out of curiosity, how would you go about doing this?
3. I think this would be really difficult to accomplish.
4. I agree if we could somehow figure out 2 or 3, that would be great.
1) As noted by Michael in a sibling comment - and I've had this happen in real life - a PID from a command which exited abnormally (and thus didn't clean up the file) can be picked up by another process, particularly on busy boxes. If your pid gets picked up by, say, an nginx worker, in which case your cron may never start again.
2) /proc/[pid]/stat column 14 - utime. Look for this to increment with every check.
3) An update function call from within the program itself - a particular count of a single location, or the lack of an update while (2) is updating could indicate an infinite loop.
The point for #1 is that PIDs are reused. Just because the cron job previously started a process with a PID 473 doesn't mean that PID 473 is that same process the next time the cron job comes around to check. It's entirely possible that the original process was finished or killed and a new process started with the same PID.
Indeed, the use of PID files alone is racy for what is a glorified script (though programmatic here) that simply creates and removes a lock file. Using the cmdline as an identifier would be more reliable. Though then you can run into multiple instances, so UUID+cmdline might help?
Comments
As someone who has had to write this themselves multiple times, there are a few bits that I consider to be missing:
1) Command line verification - is the pid owned by the same type of process as is running now? PIDs are re-used, ensure it's the same (the creation time check helps, but it doesn't say anything about what process wrote it).
2) Process Hang Detection - Has the process actually consumed any CPU ticks in the last minute?
3) Infinite loop detection - Is the other process stuck processing something uselessly?
4) Killing off stuck processes - 2 or 3 true? Behead it and continue on. Optionally do some form of alerting - stderr is probably fine.
Add these, and I would personally find it much more useful.
To address your concerns:
1. I assume that it's the same type of process because by default the PID file is being written to the current working directory of the script. If you'd like, you can specify a location yourself to ensure that each type of process is grouped on one PID file.
2. Out of curiosity, how would you go about doing this?
3. I think this would be really difficult to accomplish.
4. I agree if we could somehow figure out 2 or 3, that would be great.
1) As noted by Michael in a sibling comment - and I've had this happen in real life - a PID from a command which exited abnormally (and thus didn't clean up the file) can be picked up by another process, particularly on busy boxes. If your pid gets picked up by, say, an nginx worker, in which case your cron may never start again.
2) /proc/[pid]/stat column 14 - utime. Look for this to increment with every check.
3) An update function call from within the program itself - a particular count of a single location, or the lack of an update while (2) is updating could indicate an infinite loop.
The point for #1 is that PIDs are reused. Just because the cron job previously started a process with a PID 473 doesn't mean that PID 473 is that same process the next time the cron job comes around to check. It's entirely possible that the original process was finished or killed and a new process started with the same PID.
This is why I also store the creation time to avoid reused PIDs.
Indeed, the use of PID files alone is racy for what is a glorified script (though programmatic here) that simply creates and removes a lock file. Using the cmdline as an identifier would be more reliable. Though then you can run into multiple instances, so UUID+cmdline might help?