Logs, Metrics, and Distributed Tracing
Logs capture discrete events with context; metrics aggregate numeric measurements over time; traces show a request's path across components. These signals complement one another. A trace can reveal which dependency added latency, metrics can show whether the problem affects many requests, and logs can explain a particular failure.
Use structured logs with fields such as timestamp, severity, service, environment, request ID, trace ID, and error code. Avoid high-cardinality metric labels such as raw user IDs because they can create expensive time series. Put detailed per-request identifiers in logs and traces instead.
Propagate trace context across HTTP and message boundaries. Sampling reduces overhead but must preserve enough data to investigate rare failures. Instrument meaningful business outcomes as well as infrastructure: successful checkouts, failed payment attempts, and time-to-complete jobs can be more useful than CPU alone.
- Logs, metrics, and traces answer different questions.
- Avoid high-cardinality metric labels.
- Instrument business outcomes as well as infrastructure.
Which telemetry signal is best suited to show the path and time spent across several services for one request?