Queue Fairness, Dead Letters, and Job Operations
Separate workload classes when jobs differ in cost or urgency. A small number of expensive jobs should not block thousands of short tasks. Use priority classes, weighted fairness, concurrency limits, and tenant quotas while avoiding starvation.
Dead-letter queues capture jobs that exceed retry policy, but they require triage, ownership, and safe replay tooling. Monitor oldest job age, retry count, stuck-running state, and completion SLOs. Queue depth alone cannot tell whether business work is already late.
On shutdown, workers should stop claiming new jobs and finish or release active leases safely. Reconcile expected scheduled occurrences with completed work to detect silent omissions.
Key Takeaways
- Fair scheduling protects against noisy workloads.
- Dead-letter queues need an operational recovery path.
- Observe completion age and SLOs, not only queue depth.