Chapter•60 Minutes

Items per page

4. Chunking Strategies and Context Preservation

Choose chunk size, overlap, semantic boundaries, contextual enrichment, and evaluation methods.

Why Chunk Documents?

Chunking creates smaller retrieval units from long documents. Whole-document embeddings can be too broad; tiny chunks can lose definitions, exceptions, and surrounding context. The ideal chunk contains a coherent idea and enough context to interpret it independently.

Chunking depends on source structure, query type, embedding model, and context budget. Token-based sizes align better with model limits than character counts, but semantic boundaries are often more important than a universal threshold. Evaluate rather than copying defaults.

Diagram

Implementation example

Long document -> semantic sections -> coherent chunks
              -> metadata + embeddings -> searchable index

💡 Chunking is information design, not merely splitting text at a fixed number.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What is the purpose of chunking?

Fixed-Size Windows and Overlap

Fixed-size windows are a reproducible baseline. Overlap repeats boundary text so a relevant sentence is less likely to be separated from its context. Excessive overlap increases index size and duplicate results; too little may lose relationships. Use a tokenizer-aware splitter when possible, and handle code fences and oversized paragraphs specially.

Implementation example

function chunkWithOverlap(text, size=1200, overlap=200) {
  if (overlap >= size) throw new Error("overlap must be smaller than size");
  const chunks=[];
  for (let start=0; start<text.length; start += size-overlap) {
    chunks.push(text.slice(start, Math.min(start+size,text.length)));
  }
  return chunks;
}

💡 Character chunking is illustrative; production chunkers should be token-aware and preserve source offsets.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What can excessive overlap cause?

Semantic and Structure-Aware Chunking

Structure-aware chunking respects headings, paragraphs, lists, tables, and code blocks. Hierarchical splitting first uses high-level boundaries, then subdivides oversized sections. Store parent section IDs so a small child chunk can be expanded when its context is needed.

Model-based semantic splitting can help detect topic changes but adds cost and may be inconsistent. Compare it against a simpler baseline on representative questions. Preserve exceptions and definitions with the rules they qualify.

Implementation example

{
 "documentId":"api-guide",
 "parentSectionId":"authentication",
 "headingPath":["Security","Authentication","Token expiry"],
 "chunkIndex":4
}

💡 Parent references enable contextual expansion and source traceability.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why keep a parent section ID?

Contextual Enrichment and Parent-Child Retrieval

A chunk may need a title, product name, or section heading to be understandable. Contextual enrichment adds source-derived metadata or a short prefix; keep generated enrichment distinct from original text and version it. Parent-child retrieval indexes small chunks but can return a larger parent section for generation.

Expansion improves context but may reintroduce irrelevant text and consume more tokens. Selectively include neighbors or parent content, and preserve exact source offsets for citations.

Implementation example

Search -> matching child chunk -> load parent/neighbor context
      -> trim to relevant evidence -> generate with citations

💡 Enrichment is a retrieval aid, not a replacement for source evidence.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why distinguish enrichment from source text?

Evaluate Chunking Choices

Run controlled experiments across chunk size, overlap, boundary rules, and parent expansion. Keep the embedding model, query set, and retrieval settings fixed where possible. Measure Recall@K, context precision, answer support, latency, and index size. Inspect critical failure cases, because average scores can hide lost exceptions.

Store chunker version and parameters with the index. A chunker change generally requires reprocessing and reindexing.

Implementation example

{
 "experiment":"chunk-ablation",
 "sizesTokens":[300,600,900],
 "overlapTokens":[50,100],
 "metrics":["recall_at_5","context_precision","answer_support","latency"]
}

💡 Versioned experiment records make comparisons reproducible.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why keep the embedding model fixed during a chunking experiment?

Why Chunk Documents?

Fixed-Size Windows and Overlap

Semantic and Structure-Aware Chunking

Contextual Enrichment and Parent-Child Retrieval

Evaluate Chunking Choices

STAY UPDATED WITH BYTES!