HTML, Markdown, and Web Content
HTML ingestion should remove navigation and boilerplate without discarding substantive content. Markdown parsing should preserve heading hierarchy, code fences, tables, lists, and links. Record canonical URL, title, crawl time, language, and heading path. Canonicalization helps avoid indexing the same article under multiple URLs.
Do not flatten structured content into an undifferentiated string if structure affects meaning. Headings can be attached to chunks to make a passage interpretable when retrieved alone.
Implementation example
{
"title":"Connection pooling",
"headingPath":["Node.js","Database access","Connection pooling"],
"text":"A pool reuses database connections...",
"contentType":"markdown"
}💡 Heading paths and source metadata improve retrieval and citation traceability.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why preserve heading hierarchy?