Q43: How do you prevent duplicate jobs in a distributed worker system?

Difficulty:AdvancedTopic:Background Jobs

Assume a job may be delivered more than once. Give each job a stable ID, make the effect idempotent, and store completion or deduplication state with a uniqueness constraint. Use leases or visibility timeouts so abandoned work can be reclaimed, and ensure a worker that loses its lease cannot overwrite newer work—often by using fencing tokens or version checks. External side effects may require provider idempotency keys and reconciliation.

Key Takeaways
  • At-least-once job delivery is common; make effects idempotent.
  • Leases need expiry and protection against stale workers.