Observability, Reliability, and Support
Telemetry, service objectives, runbooks, incident coordination, and customer communication
Logs, Metrics, and Traces
Logs describe events, metrics summarize behavior, and traces connect work across services.
How to apply it
Use structured logs with timestamps, severity, request IDs, trace IDs, and safe context. Monitor request rate, errors, latency percentiles, saturation, queue depth, worker failures, and customer-facing success rates. Propagate trace context through jobs.
Worked example
A distributed trace reveals that a slow API request spends most of its time in a database query.
Common mistakes to avoid
Logging secrets or measuring CPU while ignoring customer outcomes.
Practice task
Implement this concept in the capstone. Add one positive test, one negative or failure-path test, and a short note explaining the trade-off you chose.
- Logs describe events, metrics summarize behavior, and traces connect work across services.
- Use structured logs with timestamps, severity, request IDs, trace IDs, and safe context.
What is the strongest production practice for Logs, Metrics, and Traces?
SLOs, Alerts, and Runbooks
An alert should prompt action; an SLI is a measurement, an SLO a target, and an SLA a contractual commitment.
How to apply it
Define eligible traffic and measurement windows. Use error budgets to inform reliability trade-offs. Runbooks include symptoms, dashboards, safe commands, rollback, escalation, and recovery verification.
Worked example
A 99.9% time-based availability target allows roughly 43 minutes of unavailability in a 30-day month; request-based SLOs require request-based calculations.
Common mistakes to avoid
Alerts with no owner or never testing recovery procedures.
Practice task
Implement this concept in the capstone. Add one positive test, one negative or failure-path test, and a short note explaining the trade-off you chose.
- An alert should prompt action; an SLI is a measurement, an SLO a target, and an SLA a contractual commitment.
- Define eligible traffic and measurement windows.
What is the strongest production practice for SLOs, Alerts, and Runbooks?
Incident Response and Customer Support
Prepare severity levels, incident roles, containment, evidence preservation, customer communications, recovery criteria, and post-incident reviews.
How to apply it
Coordinate an incident lead, technical responders, and communications owner as needed. Share confirmed impact and next update time. Support intake includes tenant, time, request ID, expected behavior, and actual behavior—never passwords.
Worked example
A status update names affected service, impact, mitigation, and next update time without promising unverified recovery.
Common mistakes to avoid
Blaming individuals, making uncoordinated production changes, or requesting secrets in tickets.
Practice task
Implement this concept in the capstone. Add one positive test, one negative or failure-path test, and a short note explaining the trade-off you chose.
- Prepare severity levels, incident roles, containment, evidence preservation, customer communications, recovery criteria, and post-incident reviews.
- Coordinate an incident lead, technical responders, and communications owner as needed.
What is the strongest production practice for Incident Response and Customer Support?