Latency Budgets
End-to-end latency includes query rewriting, embedding, retrieval, reranking, context assembly, model first-token time, generation, and validation. Measure p50, p95, and p99. Streaming may improve perceived responsiveness even if total completion time is similar. Instrument each stage and optimize the dominant bottleneck rather than guessing.
Implementation example
Total = rewrite + embed + retrieve + rerank
+ assemble context + generate + validate💡 Track time to first token separately from completed-answer latency.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why measure p95 latency?