If it works correctly, wouldn't those still be peak times? Except with this they have to process the initial scheduling request in addition to the at-execution task.
Everyone else's crons, synced to wall clocks, vs your centralized cron (task scheduler, really) that is aware of scheduled work and current load on your systems that consume the scheduled tasks.
Controlling the ability to nudge the wakeup times by small amounts of time can make a huge difference to your ability to manage spiky workloads like this.
A lot of answers don't go stale for hours or days. They'll do the task early, at an off-peak time, hidden from the user, double-check that it really wasn't time sensitive, then surface the saved answer at the time desired.
Start with a regex (or fast tiny model) to flag obvious time-sensitive tasks. Else, do the task early by prompting it "if this requires up to the minute information, output cancel, else [prompt]". At best, it's 1 regex + 1 full inference. At worst, it's 1 regex + 1 output token + 1 full inference.
Comments
sounds like theyre trying to get ahead of cron job wrappers so they dont get slammed at peak times
If it works correctly, wouldn't those still be peak times? Except with this they have to process the initial scheduling request in addition to the at-execution task.
Everyone else's crons, synced to wall clocks, vs your centralized cron (task scheduler, really) that is aware of scheduled work and current load on your systems that consume the scheduled tasks.
Controlling the ability to nudge the wakeup times by small amounts of time can make a huge difference to your ability to manage spiky workloads like this.
A lot of answers don't go stale for hours or days. They'll do the task early, at an off-peak time, hidden from the user, double-check that it really wasn't time sensitive, then surface the saved answer at the time desired.
How are they going to double check without incurring the cost of running it again?
Start with a regex (or fast tiny model) to flag obvious time-sensitive tasks. Else, do the task early by prompting it "if this requires up to the minute information, output cancel, else [prompt]". At best, it's 1 regex + 1 full inference. At worst, it's 1 regex + 1 output token + 1 full inference.