Distributed Schedulers and Durable Workflow Engines
Design recurring jobs, worker ownership, durable progress, retry classification, compensation, and fair queue execution.
Leases, Fencing, and Recurring Job Semantics
A distributed scheduler assigns due jobs across workers. Multiple schedulers can coordinate with leases or atomic job claims, but a paused worker may resume after its lease expires. Fencing tokens, job versions, and idempotent effects help prevent stale owners from corrupting state.
Use stable job IDs and occurrence IDs to deduplicate retries. Define what happens when a scheduled occurrence is delayed: catch up all missed runs, run once, or skip stale occurrences. Those semantics affect billing, reporting, notifications, and maintenance jobs.
Monitor scheduled-versus-started gaps, oldest due job age, duplicate claims, and stuck leases. A scheduler that appears healthy but silently omits recurring work is not reliable.
- Leases need stale-owner protection.
- Recurring jobs require explicit misfire policies.
- Track expected work against actual execution.
Durable Workflows, Retries, and Compensation
A durable workflow persists progress so execution can resume after a process or infrastructure failure. Each step needs input/output state, timeout, retry classification, idempotency strategy, and compensation behavior where relevant.
A timeout may leave an external side effect in an unknown state. Query the external system or reuse an idempotency key before repeating the operation. Retry transient network failures differently from invalid input or permanent business rejection. Use bounded backoff with jitter.
Long-running workflows need cancellation rules, versioned definitions, operator intervention, and visibility into stuck states. Replaying a workflow after code changes must not reinterpret old state unsafely. Keep large payloads outside workflow history and reference them by stable identifiers.
- Persist progress and classify retryable errors.
- Timeouts can leave side effects ambiguous.
- Workflow replay and versioning need deliberate design.
Queue Fairness, Dead Letters, and Job Operations
Separate workload classes when jobs differ in cost or urgency. A small number of expensive jobs should not block thousands of short tasks. Use priority classes, weighted fairness, concurrency limits, and tenant quotas while avoiding starvation.
Dead-letter queues capture jobs that exceed retry policy, but they require triage, ownership, and safe replay tooling. Monitor oldest job age, retry count, stuck-running state, and completion SLOs. Queue depth alone cannot tell whether business work is already late.
On shutdown, workers should stop claiming new jobs and finish or release active leases safely. Reconcile expected scheduled occurrences with completed work to detect silent omissions.
- Fair scheduling protects against noisy workloads.
- Dead-letter queues need an operational recovery path.
- Observe completion age and SLOs, not only queue depth.