Design a Chunk Schema
Each chunk needs a stable ID, source document ID, source version, text, embedding, source location, tenant/ACL metadata, and transformation versions. Stable IDs should account for document revisions and chunk boundary changes. Stale chunks must be removed or superseded on reindexing.
Metadata fields should reflect real filters and citation needs. Too few fields make authorization and provenance difficult; too many unconstrained fields add complexity.
Implementation example
{
"chunkId":"doc-18:v3:chunk-07",
"documentId":"doc-18",
"documentVersion":3,
"tenantId":"tenant-42",
"chunkerVersion":"semantic-v2",
"embeddingModel":"embedding-v1"
}💡 Versioned records support updates, citations, and reproducible retrieval.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why include document version on a chunk?