Caching, Batching, and Prompt Reuse

Caching can reduce latency and cost when inputs or outputs are safely reusable. Cache keys must include all factors that affect correctness: tenant, permissions, prompt version, model, retrieval corpus version, and relevant user input. Never share cached private answers across users or tenants. Use TTLs and invalidation when underlying data changes.

Batching can improve throughput for independent tasks but may increase latency for individual users. Use bounded concurrency and provider batch facilities when their latency and availability trade-offs fit. Prompt caching features are provider-specific; follow current provider documentation for eligible content, cache lifetime, and billing behavior.

Permission-aware cache key

const cacheKey = hash({
  tenantId,
  userScope,
  promptVersion,
  modelId,
  corpusVersion,
  normalizedInput
});

💡 A cache key must represent the scope and versions that affect the answer. The hash function is illustrative; store cache entries with expiry and ensure access checks still run before returning private results.

Key Takeaways

Cache only when reuse is correct and permission-safe.
Include prompt/model/corpus versions in cache identity.
Use batching only when latency requirements allow it.

Knowledge Check

Why should tenant or permission scope be represented in a cache key?

Caching, Batching, and Prompt Reuse

STAY UPDATED WITH BYTES!