Ingestion: Parsing, Chunking, Deduplication, and Reindexing
Ingestion should be idempotent: running it twice should not create duplicate chunks or stale copies. Store a stable document ID, content hash, source URI, version, parser version, chunking configuration, and embedding model ID. When content changes, delete or supersede prior chunks consistently and ensure failed ingestion jobs can resume.
Different file types require different parsers. Preserve headings and page/section provenance for PDFs and office documents; extract code without destroying indentation; handle OCR uncertainty explicitly. Validate file size and type, scan uploads where appropriate, and treat document contents as untrusted.
Ingestion record example
{
"documentId": "policy-returns-v4",
"contentHash": "sha256:...",
"source": "internal://policies/returns",
"version": 4,
"parserVersion": "pdf-parser-2",
"embeddingModel": "embedding-model-id",
"status": "indexed"
}💡 Metadata makes reprocessing, deletion, and debugging traceable. The hash is a placeholder; compute it from normalized content using a cryptographic hash function in your application.
Key Takeaways
- Make ingestion idempotent and resumable.
- Preserve source provenance and parser version.
- Plan for deletion and reindexing when content changes.
Knowledge Check
Why should an ingestion pipeline record document and embedding versions?