Faithfulness, Relevance, and Citation Quality
Evaluate whether claims are supported by context, whether the answer addresses the question, and whether citations support each claim. Automated graders scale evaluation but can share generator biases; calibrate them against human ratings. Deterministic checks can verify that cited IDs exist and were retrieved.
Evaluate retrieval and generation separately: good retrieval plus unsupported generation points to a different fix than missing evidence.
Implementation example
Retrieval: relevant chunks found?
Grounding: claims supported?
Relevance: answers the question?
Citation: source supports claim?
Policy: answer permitted and safe?💡 Separate dimensions help localize failure.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
What does citation correctness evaluate?