Deduplication, Diversity, and Contradictions
Near-duplicate chunks can crowd out useful evidence. Deduplicate by source/content similarity and consider diversity across sections, but do not force diversity when one authoritative source suffices. Contradictions may reflect stale versions, effective-date boundaries, or genuine disagreement. Preserve authority and timestamps and disclose unresolved conflicts.
Evaluate whether diversity improves coverage without removing unique exceptions.
Implementation example
Selection goals:
- relevance and source authority
- coverage of subquestions
- low redundancy
- freshness
- preservation of exceptions and conflicts💡 Diversity is useful only when it improves coverage without losing critical evidence.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why not always select chunks from different documents?