Queues, Retries, and Backpressure
Use durable queues, bounded concurrency, exponential backoff with jitter, checkpoints, and idempotent upserts. Respect provider retry hints and distinguish transient errors from invalid input. Backpressure protects parsers, embedding APIs, and indexes. Quarantine poison documents instead of retrying forever. Keep ingestion capacity separate from interactive query capacity.
Implementation example
async function processJob(job) {
try { await ingest(job); await queue.ack(job.id); }
catch (e) {
if (isRetryable(e)) await queue.retry(job.id, backoff(job.attempt));
else await queue.deadLetter(job.id, safeError(e));
}
}💡 Production queues need visibility timeouts, retry caps, idempotency, and graceful shutdown.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why bound worker concurrency?