Normalization and Deduplication
Normalize whitespace and encoding while preserving headings, lists, tables, code, units, and legal qualifications. Exact hashes catch identical content; normalized hashes catch formatting-only differences. Near-duplicate detection should preserve meaningful historical versions.
Version normalization logic. A parser or cleaner change may require reprocessing. Store raw source references and normalized output so a retrieval failure can be traced back to the original.
Implementation example
function normalizeText(text) {
return text.replace(/\r\n/g, "\n")
.replace(/[\t ]+/g, " ")
.replace(/\n{3,}/g, "\n\n")
.trim();
}💡 This is a conservative baseline; format-aware parsers must preserve semantic structure.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
What is a risk of aggressive normalization?