Partial Failure, Timeouts, and Deadlines

In a distributed system, one component can fail while others continue. A client may time out even though the server completed the request; a response may be lost after a database commit. Therefore, “no response” is not the same as “no operation occurred.”

Use bounded connection, request, and total deadlines. A deadline expresses how long the caller can still wait; propagate the remaining budget downstream. Retries should be limited, use exponential backoff with jitter, and apply only when the operation is safe to retry or protected by idempotency.

Retries can worsen overload if every layer retries independently. If three layers each retry four times, one user request may trigger many downstream attempts. Choose one retry owner where possible, cap attempts, and stop retrying when the deadline or circuit-breaker policy says further work is harmful.

Key Takeaways
  • Design for partial failure as a normal condition.
  • Timeouts need explicit retry and idempotency policies.
  • Uncoordinated retries can multiply load during an outage.
Knowledge Check

A client times out after submitting a request. What can safely be concluded?