Chunking, Overlap, and Metadata
Chunking splits long documents into retrievable units. Very large chunks can dilute relevance and exceed context budgets; very small chunks can lose context. Chunk by semantic structure where possible—headings, paragraphs, sections, code blocks—then tune chunk size and overlap using evaluation data.
Metadata should include source, document version, tenant or access scope, section heading, timestamps, and content type. Apply authorization filters before returning chunks to the model. A vector similarity score does not imply permission to disclose a document.
Chunk a document by paragraphs
function chunkParagraphs(text, maxChars = 1200) {
const paragraphs = text.split(/\n\s*\n/);
const chunks = [];
let current = "";
for (const paragraph of paragraphs) {
if (current && (current.length + paragraph.length + 2) > maxChars) {
chunks.push(current);
current = "";
}
current += (current ? "\n\n" : "") + paragraph;
}
if (current) chunks.push(current);
return chunks;
}💡 This simple baseline groups paragraphs by character count. Production chunking should consider token limits, oversized paragraphs, headings, overlap, code blocks, and document structure.
Key Takeaways
- Chunk size and overlap should be evaluated, not guessed permanently.
- Preserve source and version metadata.
- Enforce access control before model context assembly.
Knowledge Check
Why store access-control metadata with indexed chunks?