Token Budgets and Cost Modeling
Cost depends on provider pricing, model choice, input/output token volume, modalities, tool usage, caching, and retries. Estimate per-request cost from measured usage and multiply by expected volume, retry rate, and background jobs. Include evaluation, indexing, embeddings, and operational overhead in the product budget.
Set separate budgets for interactive requests and batch jobs. Cap maximum output and context size; retrieve only useful chunks; avoid repeated model calls when deterministic code can answer. Monitor actual usage because prompt changes and user behavior can shift costs unexpectedly.
Cost estimate pseudocode
function estimateCost({ inputTokens, outputTokens, inputRate, outputRate }) {
return (inputTokens / 1_000_000) * inputRate
+ (outputTokens / 1_000_000) * outputRate;
}💡 Rates are provider- and model-specific and may change. Treat this as a calculation pattern; load current rates from a maintained configuration and reconcile estimates with actual usage records.
Key Takeaways
- Measure real token usage and retries.
- Separate interactive and batch budgets.
- Revisit cost assumptions as models and pricing change.
Knowledge Check
Which is a reliable way to reduce unnecessary model cost?