Leases, Fencing, and Recurring Job Semantics

A distributed scheduler assigns due jobs across workers. Multiple schedulers can coordinate with leases or atomic job claims, but a paused worker may resume after its lease expires. Fencing tokens, job versions, and idempotent effects help prevent stale owners from corrupting state.

Use stable job IDs and occurrence IDs to deduplicate retries. Define what happens when a scheduled occurrence is delayed: catch up all missed runs, run once, or skip stale occurrences. Those semantics affect billing, reporting, notifications, and maintenance jobs.

Monitor scheduled-versus-started gaps, oldest due job age, duplicate claims, and stuck leases. A scheduler that appears healthy but silently omits recurring work is not reliable.

Key Takeaways
  • Leases need stale-owner protection.
  • Recurring jobs require explicit misfire policies.
  • Track expected work against actual execution.