Evaluate Conversational Retrieval
Test follow-ups, pronouns, topic switches, corrections, multiple active documents, and earlier citations. Measure rewrite fidelity, retrieval quality, and answer quality separately. Fluent rewrites can still change intent. Do not rewrite self-contained questions unnecessarily.
Implementation example
{
"priorContext":"Question about Pro plan storage",
"followUp":"Does it include backups?",
"expectedQuery":"Does the Pro plan include backups?"
}💡 A full test set should include ambiguous cases where clarification is correct.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
What should rewrite evaluation measure?