Cost Modeling and Budgets
Costs include parsing/OCR, embeddings, storage, search, reranking, model input/output, evaluation, observability, retries, and reindexing. Estimate per-query and per-document costs, then model realistic traffic. Reduce unnecessary context, cache safely, batch work, cap outputs, and use smaller models only when evaluation supports the choice. Keep provider rates in maintained configuration and reconcile estimates with actual usage.
Implementation example
Per query = query embedding + search + rerank
+ model input/output + retries + observability💡 Include retries, reindexing, and evaluation in budgets.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Which is a sensible optimization?