The problem is that this doesn't address what I have found to be the most dangerous problem with periodic tasks.
That is when your cron job runs but there is some error in part of the script (for example maybe it writes/reads a file in a folder but the permissions on that folder were changed since the script was written). This causes an error which might cause a cascade of errors meaning that some other parts of your job either fail to run or run incorrectly.
Now what happens here, do you get notified of the error or does it just get silently eaten? It's also very possible that your system will eat the error and then proceed to the next step (calling this API) and everything will appear fine.
One thing I figure out what the expected output from the job should be, I then pipe the output from the cronjob into a file. I have a second cronjob that checks the contents of this file periodically and generates an alert if it does not match what is expected.
You should also try and find some way to test any generated data. For example if you are doing a DB backup, add another table with a field that contains data that is in some way based on the date. You can then have a task which will try and restore old backups into another DB, it can then check this field against the expected value for the date of the backup.
Of course none of these techniques are silver bullets and there are plenty of things that can go wrong, it is certainly prudent to check things manually every once in a while.
Perhaps this API could be modified to take as an input the output from scheduled tasks and check them?
You have to make sure your periodic tasks acts in such a way that if a piece fails, the rest doesn't execute (using && for instance). Then you would add hitting your special url the last thing to execute.
If you have a way of checking things like this, great! You should keep doing that!
That is true, but makes the assumption that all scripts along the way will return the correct values on errors etc. Not always true for rushed in-house scripts.
You may also have to consider warning conditions which might be more catastrophic to your "pipe" that the original program would believe.
Comments
The problem is that this doesn't address what I have found to be the most dangerous problem with periodic tasks.
That is when your cron job runs but there is some error in part of the script (for example maybe it writes/reads a file in a folder but the permissions on that folder were changed since the script was written). This causes an error which might cause a cascade of errors meaning that some other parts of your job either fail to run or run incorrectly.
Now what happens here, do you get notified of the error or does it just get silently eaten? It's also very possible that your system will eat the error and then proceed to the next step (calling this API) and everything will appear fine.
One thing I figure out what the expected output from the job should be, I then pipe the output from the cronjob into a file. I have a second cronjob that checks the contents of this file periodically and generates an alert if it does not match what is expected.
You should also try and find some way to test any generated data. For example if you are doing a DB backup, add another table with a field that contains data that is in some way based on the date. You can then have a task which will try and restore old backups into another DB, it can then check this field against the expected value for the date of the backup.
Of course none of these techniques are silver bullets and there are plenty of things that can go wrong, it is certainly prudent to check things manually every once in a while.
Perhaps this API could be modified to take as an input the output from scheduled tasks and check them?
You have to make sure your periodic tasks acts in such a way that if a piece fails, the rest doesn't execute (using && for instance). Then you would add hitting your special url the last thing to execute.
If you have a way of checking things like this, great! You should keep doing that!
That is true, but makes the assumption that all scripts along the way will return the correct values on errors etc. Not always true for rushed in-house scripts.
You may also have to consider warning conditions which might be more catastrophic to your "pipe" that the original program would believe.