15. Cost, Latency, Caching, and Scaling

Budget AI workloads, optimize prompts and retrieval, handle rate limits, and scale background processing.

Token Budgets and Cost Modeling

Cost depends on provider pricing, model choice, input/output token volume, modalities, tool usage, caching, and retries. Estimate per-request cost from measured usage and multiply by expected volume, retry rate, and background jobs. Include evaluation, indexing, embeddings, and operational overhead in the product budget.

Set separate budgets for interactive requests and batch jobs. Cap maximum output and context size; retrieve only useful chunks; avoid repeated model calls when deterministic code can answer. Monitor actual usage because prompt changes and user behavior can shift costs unexpectedly.

Cost estimate pseudocode

function estimateCost({ inputTokens, outputTokens, inputRate, outputRate }) {
  return (inputTokens / 1_000_000) * inputRate
       + (outputTokens / 1_000_000) * outputRate;
}

💡 Rates are provider- and model-specific and may change. Treat this as a calculation pattern; load current rates from a maintained configuration and reconcile estimates with actual usage records.

Key Takeaways

Measure real token usage and retries.
Separate interactive and batch budgets.
Revisit cost assumptions as models and pricing change.

Knowledge Check

Which is a reliable way to reduce unnecessary model cost?

Caching, Batching, and Prompt Reuse

Caching can reduce latency and cost when inputs or outputs are safely reusable. Cache keys must include all factors that affect correctness: tenant, permissions, prompt version, model, retrieval corpus version, and relevant user input. Never share cached private answers across users or tenants. Use TTLs and invalidation when underlying data changes.

Batching can improve throughput for independent tasks but may increase latency for individual users. Use bounded concurrency and provider batch facilities when their latency and availability trade-offs fit. Prompt caching features are provider-specific; follow current provider documentation for eligible content, cache lifetime, and billing behavior.

Permission-aware cache key

const cacheKey = hash({
  tenantId,
  userScope,
  promptVersion,
  modelId,
  corpusVersion,
  normalizedInput
});

💡 A cache key must represent the scope and versions that affect the answer. The hash function is illustrative; store cache entries with expiry and ensure access checks still run before returning private results.

Key Takeaways

Cache only when reuse is correct and permission-safe.
Include prompt/model/corpus versions in cache identity.
Use batching only when latency requirements allow it.

Knowledge Check

Why should tenant or permission scope be represented in a cache key?

Rate Limits, Retries, and Circuit Breakers

Provider rate limits may constrain requests, tokens, concurrent jobs, or organization-wide usage. Respect retry hints such as Retry-After when present, use exponential backoff with jitter, and cap total elapsed time. A circuit breaker can temporarily stop calls to a failing dependency, allowing recovery and preventing a cascading backlog.

Do not retry all failures. Authentication errors, invalid schemas, policy denials, and unsupported model requests usually require a code or input change. Handle partial success in batch jobs and persist progress so a transient failure does not restart the whole batch.

Simple circuit breaker states

CLOSED: requests flow normally
  -- repeated failures -->
OPEN: fail fast during cooldown
  -- cooldown elapsed -->
HALF_OPEN: allow limited probe
  -- success --> CLOSED
  -- failure --> OPEN

💡 Circuit breakers protect the rest of the application during a provider outage. Choose thresholds and cooldowns based on measured traffic and avoid hiding persistent failures from operators.

Key Takeaways

Retry transient errors only.
Use jitter and honor provider retry guidance.
Use circuit breakers and durable queues for resilience.

Knowledge Check

What is the purpose of a circuit breaker?

Token Budgets and Cost Modeling

Caching, Batching, and Prompt Reuse

Rate Limits, Retries, and Circuit Breakers

STAY UPDATED WITH BYTES!