Fixed-Size Windows and Overlap
Fixed-size windows are a reproducible baseline. Overlap repeats boundary text so a relevant sentence is less likely to be separated from its context. Excessive overlap increases index size and duplicate results; too little may lose relationships. Use a tokenizer-aware splitter when possible, and handle code fences and oversized paragraphs specially.
Implementation example
function chunkWithOverlap(text, size=1200, overlap=200) {
if (overlap >= size) throw new Error("overlap must be smaller than size");
const chunks=[];
for (let start=0; start<text.length; start += size-overlap) {
chunks.push(text.slice(start, Math.min(start+size,text.length)));
}
return chunks;
}💡 Character chunking is illustrative; production chunkers should be token-aware and preserve source offsets.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
What can excessive overlap cause?