Ingestion Quality Gates and Dead Letters
Track states such as queued, parsing, chunking, embedding, indexed, failed, and quarantined. Validate minimum extracted text, metadata, page count, and duplicate rates. Retry transient network failures; quarantine corrupt files or repeatedly failing records. Keep safe error summaries and an operator path to inspect, retry, or remove documents.
A document should not be marked searchable until all required stages complete successfully.
Implementation example
queued -> parsing -> chunking -> embedding -> indexed
\-> retryable failure -> retry
\-> permanent failure -> dead-letter💡 A state machine makes partial failures visible and recoverable.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why use a dead-letter queue?