3 Topics
Chapter•115 Minutes

Reliability Engineering and Disaster Recovery

Learn availability targets, failure isolation, retry policies, graceful degradation, and recovery planning.

SLIs, SLOs, SLAs, and Error Budgets

An SLI is a measured indicator such as successful request ratio or latency. An SLO is a target for that indicator over a time window. An SLA is an agreement with consequences if commitments are missed. Keep these terms distinct.

For a 30-day month, 99.9% availability permits about 43.2 minutes of unavailability if measured continuously using that simple model; 99.99% permits about 4.32 minutes. The real calculation depends on the SLI definition, excluded events, and measurement method.

An error budget is the allowed unreliability implied by an SLO. Teams can use it to balance feature delivery and reliability work. Do not optimize availability at any cost: the goal is a level that meets user needs while preserving sustainable delivery and cost.

Key Takeaways
  • Define SLIs before setting SLO targets.
  • Availability percentages imply concrete downtime budgets.
  • Error budgets make reliability trade-offs explicit.
Knowledge Check

What does an RPO define?

Retries, Circuit Breakers, Bulkheads, and Fallbacks

Retries recover from transient failures, but must be bounded and safe. Exponential backoff increases the delay after each failed attempt; jitter spreads retries so clients do not synchronize into a retry storm.

A circuit breaker temporarily stops calls to a dependency after a failure threshold, allowing recovery and preventing repeated work against an unhealthy service. It typically moves among closed, open, and half-open states. Thresholds and probe behavior should be tuned from observed traffic and failure patterns.

Bulkheads isolate resources—for example, separate connection pools or concurrency limits for different dependencies—so one slow service does not consume every worker. Fallbacks should be semantically safe: serving a stale product description may be acceptable, while inventing a successful payment result is not.

Circuit breaker states
Key Takeaways
  • Retries, breakers, and bulkheads solve different failure problems.
  • Fallbacks must preserve business correctness.
  • Test recovery and half-open behavior, not only failure detection.

Backups, RTO, RPO, and Disaster Recovery

Recovery Time Objective (RTO) is the target time to restore service. Recovery Point Objective (RPO) is the acceptable amount of data loss measured in time. A service may have a short RTO but a large RPO if it can be restored quickly from an older backup.

Backups must be isolated from the failure modes they protect against, encrypted, access-controlled, and tested. A backup that has never been restored is an unverified assumption. Include point-in-time recovery, retention policy, deletion protection, and recovery of configuration, keys, and dependencies.

Disaster recovery strategies range from backup-and-restore to pilot light, warm standby, and active-active regions. Cost and complexity increase as recovery becomes faster. Exercise region loss, credential compromise, accidental deletion, and corrupted data scenarios; failover alone does not fix bad data replicated everywhere.

Key Takeaways
  • Define RTO and RPO with product owners.
  • Test restores and complete recovery workflows regularly.
  • Regional redundancy does not replace backups or corruption recovery.