Production Incident Reasoning
When production fails, start with symptoms rather than assumptions. Identify when the problem began, whether all users are affected, which routes or regions are affected, and whether a deployment or infrastructure change occurred.
Then inspect error rates, latency, logs, dependency health, database health, and recent changes.
A strong engineer avoids making several unrelated changes simultaneously because that makes the root cause harder to identify.
Key Takeaways
- Start incident debugging with observable symptoms.
- Correlate failures with recent deployments and infrastructure changes.
- Change one major variable at a time when possible.