Rate Limits, Retries, and Circuit Breakers
Provider rate limits may constrain requests, tokens, concurrent jobs, or organization-wide usage. Respect retry hints such as Retry-After when present, use exponential backoff with jitter, and cap total elapsed time. A circuit breaker can temporarily stop calls to a failing dependency, allowing recovery and preventing a cascading backlog.
Do not retry all failures. Authentication errors, invalid schemas, policy denials, and unsupported model requests usually require a code or input change. Handle partial success in batch jobs and persist progress so a transient failure does not restart the whole batch.
Simple circuit breaker states
CLOSED: requests flow normally
-- repeated failures -->
OPEN: fail fast during cooldown
-- cooldown elapsed -->
HALF_OPEN: allow limited probe
-- success --> CLOSED
-- failure --> OPEN💡 Circuit breakers protect the rest of the application during a provider outage. Choose thresholds and cooldowns based on measured traffic and avoid hiding persistent failures from operators.
Key Takeaways
- Retry transient errors only.
- Use jitter and honor provider retry guidance.
- Use circuit breakers and durable queues for resilience.
Knowledge Check
What is the purpose of a circuit breaker?