A lot of answers don't go stale for hours or days. They'll do the task early, at an off-peak time, hidden from the user, double-check that it really wasn't time sensitive, then surface the saved answer at the time desired.
Start with a regex (or fast tiny model) to flag obvious time-sensitive tasks. Else, do the task early by prompting it "if this requires up to the minute information, output cancel, else [prompt]". At best, it's 1 regex + 1 full inference. At worst, it's 1 regex + 1 output token + 1 full inference.
Comments
A lot of answers don't go stale for hours or days. They'll do the task early, at an off-peak time, hidden from the user, double-check that it really wasn't time sensitive, then surface the saved answer at the time desired.
How are they going to double check without incurring the cost of running it again?
Start with a regex (or fast tiny model) to flag obvious time-sensitive tasks. Else, do the task early by prompting it "if this requires up to the minute information, output cancel, else [prompt]". At best, it's 1 regex + 1 full inference. At worst, it's 1 regex + 1 output token + 1 full inference.