Build a Gold Evaluation Dataset
Each evaluation case should record a query, relevant source IDs, expected answer properties, acceptable alternatives, and whether the correct behavior is to abstain or clarify. Include common queries, critical rare policies, ambiguous follow-ups, multilingual inputs, and adversarial examples. Have domain experts review the set and add privacy-safe real failures.
Version the dataset and separate tuning examples from held-out regression cases to avoid overfitting.
Implementation example
{
"caseId":"refund-ambiguous-07",
"query":"Can I return an opened item?",
"relevantSources":["returns-policy-v4"],
"expectedBehavior":"Ask for product category if needed",
"mustNot":["Invent a universal rule"]
}💡 Expected behavior and evidence are more useful than one exact wording.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why include expected source IDs?