15. Performance and Cost Optimization

Budget latency, batch embeddings, cache safely, and optimize cost without sacrificing correctness.

Latency Budgets

End-to-end latency includes query rewriting, embedding, retrieval, reranking, context assembly, model first-token time, generation, and validation. Measure p50, p95, and p99. Streaming may improve perceived responsiveness even if total completion time is similar. Instrument each stage and optimize the dominant bottleneck rather than guessing.

Implementation example

Total = rewrite + embed + retrieve + rerank
      + assemble context + generate + validate

💡 Track time to first token separately from completed-answer latency.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why measure p95 latency?

Batch Embeddings and Reuse Work

Batching can reduce network overhead during ingestion. Use bounded concurrency, persist progress, and deduplicate unchanged content before embedding. Cache vectors only when normalized content hash, embedding model, and preprocessing version match. Larger batches can increase memory and failure blast radius, so benchmark realistic workloads.

Implementation example

const key = hash({
 contentHash: sha256(normalizedText),
 modelId: embeddingModelId,
 preprocessingVersion: "normalize-v3"
});

💡 The key must include all factors affecting vector compatibility.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What belongs in an embedding cache key?

Query and Answer Caching

RAG answers depend on permissions, corpus version, prompt version, model, and current data. Cache keys should include relevant scope and versions. Re-check authorization before returning private cached content. Use TTLs and invalidation on source updates, permission changes, or reindexing. Caching retrieval results may be safer than caching final answers for dynamic data.

Implementation example

const key = hash({
 tenantId: auth.tenantId,
 userScope: auth.cacheScope,
 query: normalizeQuery(query),
 corpusVersion, promptVersion, modelId
});

💡 The key is illustrative; include every state variable that affects correctness.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why include corpus version in a cache key?

Cost Modeling and Budgets

Costs include parsing/OCR, embeddings, storage, search, reranking, model input/output, evaluation, observability, retries, and reindexing. Estimate per-query and per-document costs, then model realistic traffic. Reduce unnecessary context, cache safely, batch work, cap outputs, and use smaller models only when evaluation supports the choice. Keep provider rates in maintained configuration and reconcile estimates with actual usage.

Implementation example

Per query = query embedding + search + rerank
          + model input/output + retries + observability

💡 Include retries, reindexing, and evaluation in budgets.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Which is a sensible optimization?

Latency Budgets

Batch Embeddings and Reuse Work

Query and Answer Caching

Cost Modeling and Budgets

STAY UPDATED WITH BYTES!