Alerting, SLO-Based Alerts, and Incident Response

Alerts should signal actionable user impact or imminent risk, not every unusual metric. Alerting on a symptom such as elevated failed-request ratio often provides more value than paging for a CPU spike that has no effect on users. Use multi-window burn-rate alerts to detect both fast severe incidents and slower SLO erosion.

An incident process defines severity, roles, communication channels, escalation, mitigation authority, and customer updates. During an incident, stabilize first: rollback, disable a feature, shed load, or route around a failing dependency when safe. Root-cause analysis follows after impact is controlled.

Blameless postmortems focus on contributing conditions and system improvements rather than individual fault. Record impact, timeline, detection gaps, decisions, and specific follow-up owners. A postmortem without tracked actions rarely improves reliability.

Key Takeaways
  • Alert on actionable symptoms and SLO risk.
  • Mitigation and clear communication precede deep root-cause analysis.
  • Postmortem actions need owners and verification.