Q43: How do you prevent duplicate jobs in a distributed worker system?
Assume a job may be delivered more than once. Give each job a stable ID, make the effect idempotent, and store completion or deduplication state with a uniqueness constraint. Use leases or visibility timeouts so abandoned work can be reclaimed, and ensure a worker that loses its lease cannot overwrite newer work—often by using fencing tokens or version checks. External side effects may require provider idempotency keys and reconciliation.
Key Takeaways
- At-least-once job delivery is common; make effects idempotent.
- Leases need expiry and protection against stale workers.