Retry Queues Need Deadlines, Not Just Attempt Counts
“Retry five times” is a policy without a clock. Five fast failures might finish in seconds; five exponential retries might run tomorrow, after the user has cancelled the order or the deploy has been superseded.
An attempt count controls how much pressure a job can create. A deadline controls whether the job is still worth doing.
{
"job_id": "email-…",
"created_at": "2026-09-14T10:00:00Z",
"expires_at": "2026-09-14T10:30:00Z",
"attempt": 3
}
Before each retry, compare the next eligible time with expires_at. If it has passed, record an explicit expired outcome and notify the appropriate owner. Do not quietly discard it, and do not keep retrying merely because the counter has room left.
Freshness is part of correctness
Some jobs can run late safely: a nonurgent search-index update may be catch-up work. Others cannot: a one-time login link, inventory reservation, or fraud decision has a clear usefulness window. Treat that difference as a product policy supplied at enqueue time, not a scheduler default guessed by the queue.
Deadlines also make incidents calmer. During an outage, operators can identify which backlog is still actionable and which should be expired or recomputed. Without that distinction, recovery often replays days of side effects in the original order and calls it reliability.
References
- Google SRE Book: Addressing Cascading Failures explains bounded retries and the harm from uncontrolled retry behavior.
- RFC 9110, section 10.2.3 defines
Retry-After, an example of time-aware retry guidance.
Final take
An operation that is no longer useful should not become more likely merely because it has been failing for longer.
Changelog
- 2026-10-03T00:08:49-04:00: Recovered the 2026-09-14 scheduled slot following archive and source review.