13. RAG Evaluation and Regression Testing

Build gold datasets, measure retrieval and groundedness, run ablations, and set release gates.

Build a Gold Evaluation Dataset

Each evaluation case should record a query, relevant source IDs, expected answer properties, acceptable alternatives, and whether the correct behavior is to abstain or clarify. Include common queries, critical rare policies, ambiguous follow-ups, multilingual inputs, and adversarial examples. Have domain experts review the set and add privacy-safe real failures.

Version the dataset and separate tuning examples from held-out regression cases to avoid overfitting.

Implementation example

{
 "caseId":"refund-ambiguous-07",
 "query":"Can I return an opened item?",
 "relevantSources":["returns-policy-v4"],
 "expectedBehavior":"Ask for product category if needed",
 "mustNot":["Invent a universal rule"]
}

💡 Expected behavior and evidence are more useful than one exact wording.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why include expected source IDs?

Recall@K, Precision@K, MRR, and nDCG

Recall@K measures relevant evidence found within the top K; Precision@K measures the share of top K that is relevant; MRR rewards early placement of the first relevant result; nDCG supports graded relevance and ranking quality. No single metric is enough. Use different metrics for coverage and noise, and inspect examples manually.

Implementation example

function recallAtK(ids,relevant,k) {
 const set=new Set(relevant);
 if(!set.size) return null;
 return ids.slice(0,k).filter(id=>set.has(id)).length/set.size;
}

💡 This simple implementation assumes IDs and relevance labels are correct.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Which metric asks whether relevant items appear within top K?

Faithfulness, Relevance, and Citation Quality

Evaluate whether claims are supported by context, whether the answer addresses the question, and whether citations support each claim. Automated graders scale evaluation but can share generator biases; calibrate them against human ratings. Deterministic checks can verify that cited IDs exist and were retrieved.

Evaluate retrieval and generation separately: good retrieval plus unsupported generation points to a different fix than missing evidence.

Implementation example

Retrieval: relevant chunks found?
Grounding: claims supported?
Relevance: answers the question?
Citation: source supports claim?
Policy: answer permitted and safe?

💡 Separate dimensions help localize failure.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What does citation correctness evaluate?

Ablation Studies

Compare lexical-only, vector-only, hybrid, reranking on/off, chunk-size variants, and prompt variants. Keep other factors fixed where possible. Report quality, latency, and cost, and slice results by query type, source, language, and risk. Averages can hide critical regressions. Version configurations and reproduce the winning setup.

Implementation example

Baseline: vector + fixed chunks
A: lexical only
B: hybrid
C: hybrid + reranker
Compare recall, support, latency, and cost.

💡 Ablations identify which components contribute measurable value.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why vary one component at a time?

Regression Gates and Release Criteria

Define release criteria such as minimum retrieval recall for critical sources, acceptable citation correctness, maximum unsupported-answer rate, latency SLO, and cost budget. Include security tests for cross-tenant access, injection, stale sources, and deletion. Maintain rollback for prompts, models, indexes, and settings.

After launch, monitor distribution shifts and update evaluation sets through review.

Implementation example

{
 "minRecallAt5":0.90,
 "minCitationSupport":0.95,
 "maxP95LatencyMs":4000,
 "securityChecksPassed":true
}

💡 Values are illustrative; set thresholds from product risk and measured baseline.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What is the purpose of an evaluation release gate?

Build a Gold Evaluation Dataset

Recall@K, Precision@K, MRR, and nDCG

Faithfulness, Relevance, and Citation Quality

Ablation Studies

Regression Gates and Release Criteria

STAY UPDATED WITH BYTES!