3. Parsing Documents and Multimodal Sources

Extract text and structure from web pages, Markdown, PDFs, scans, tables, code, and structured records.

HTML, Markdown, and Web Content

HTML ingestion should remove navigation and boilerplate without discarding substantive content. Markdown parsing should preserve heading hierarchy, code fences, tables, lists, and links. Record canonical URL, title, crawl time, language, and heading path. Canonicalization helps avoid indexing the same article under multiple URLs.

Do not flatten structured content into an undifferentiated string if structure affects meaning. Headings can be attached to chunks to make a passage interpretable when retrieved alone.

Implementation example

{
 "title":"Connection pooling",
 "headingPath":["Node.js","Database access","Connection pooling"],
 "text":"A pool reuses database connections...",
 "contentType":"markdown"
}

💡 Heading paths and source metadata improve retrieval and citation traceability.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why preserve heading hierarchy?

PDFs, OCR, and Layout

PDF extraction may fail on scans, columns, headers, footers, and tables. Scans need OCR, which can introduce errors. Preserve page numbers and, where possible, offsets or bounding boxes so evidence can be traced to the original. Inspect representative page images during ingestion QA.

Tables need row/column relationships; concatenating cells may change meaning. Evaluate parsing quality separately from retrieval and generation so extraction defects are not misdiagnosed as model hallucinations.

Implementation example

{
 "documentId":"handbook",
 "page":12,
 "section":"Leave policy",
 "text":"Eligible employees may request...",
 "extractionMethod":"ocr"
}

💡 Page provenance supports citations and helps diagnose OCR errors.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why preserve PDF page provenance?

Tables, Code, and Structured Data

Tables should carry their headers into each row representation. Code chunks should preserve indentation, symbols, language, file path, and function/class boundaries. For exact numeric fields, use a database query rather than asking a model to infer values from a flattened table.

A good system combines RAG for explanations with SQL or APIs for exact structured values. Keep the original record ID and parser version for traceability.

Implementation example

Table row:
Plan=Basic; Monthly limit=100 requests; Overage=Not available

💡 Including column headers makes an isolated row meaningful.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What is best for an exact product price?

Ingestion Quality Gates and Dead Letters

Track states such as queued, parsing, chunking, embedding, indexed, failed, and quarantined. Validate minimum extracted text, metadata, page count, and duplicate rates. Retry transient network failures; quarantine corrupt files or repeatedly failing records. Keep safe error summaries and an operator path to inspect, retry, or remove documents.

A document should not be marked searchable until all required stages complete successfully.

Implementation example

queued -> parsing -> chunking -> embedding -> indexed
                    \-> retryable failure -> retry
                    \-> permanent failure -> dead-letter

💡 A state machine makes partial failures visible and recoverable.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why use a dead-letter queue?

HTML, Markdown, and Web Content

PDFs, OCR, and Layout

Tables, Code, and Structured Data

Ingestion Quality Gates and Dead Letters

STAY UPDATED WITH BYTES!