Three pillars of observability

• Metrics: numerical time series for rates, latency, errors, saturation, and business outcomes.
• Logs: structured event records with useful context.
• Traces: request paths across services and dependencies.

Useful service indicators

• Request rate.
• Error rate, separated by meaningful error class.
• Latency percentiles, especially p95 and p99.
• Resource saturation: CPU, memory, connection pools, disk, and thread or event-loop pressure.
• Dependency latency and failure rate.
• Queue depth, oldest-message age, consumer lag, and retry counts.

SLO, SLI, and SLA

• SLI: the measured indicator, such as the proportion of eligible requests completed successfully within a latency target.
• SLO: the internal target for that indicator over a defined window.
• SLA: an external commitment, often with contractual consequences.

Logging checklist

• Use structured logs and correlation or trace IDs.
• Avoid secrets, credentials, and unnecessary personal data.
• Include meaningful error context without exposing sensitive payloads.
• Keep timestamps and service identity consistent.

Incident response

1. Detect and assess user impact.
2. Stabilize the service and reduce the blast radius.
3. Communicate status and ownership.
4. Restore service using a known safe mitigation.
5. Preserve relevant evidence and investigate the cause.
6. Conduct a blameless review and track corrective actions.

A dashboard is useful only if it helps someone identify impact, diagnose the problem, or take action. Alert on actionable symptoms and user impact rather than every noisy internal metric.