Tracing, Metrics, Cost, and Production Debugging
Record a trace across the user request, retrieval, model call, tool execution, and final response. Useful metrics include request volume, success/error rate, time to first token, total latency, token usage, estimated cost, retrieval scores, tool call counts, and user feedback. Use correlation IDs to connect events without logging unnecessary raw content.
Monitor by feature and tenant so a costly or failing workflow can be isolated. Sample or redact content logs, restrict access to traces, and define retention. Model upgrades can change quality and tool behavior, so monitor drift and rerun evaluations before broad rollout.
Structured trace record
{
"traceId": "trace_abc",
"feature": "knowledge-assistant",
"model": "configured-model",
"retrievalCount": 6,
"toolCalls": 1,
"latencyMs": 1840,
"inputTokens": 1200,
"outputTokens": 220,
"status": "success"
}💡 Use provider usage fields when available and label cost as estimated if pricing is calculated separately. Avoid recording raw prompts or outputs by default if they may contain sensitive data.
Key Takeaways
- Trace across retrieval, model, tools, and output validation.
- Track cost and latency by feature and tenant.
- Redact content and control trace access and retention.
Knowledge Check
Which is a useful production metric for streaming chat?