14. Evaluation, Testing, Observability, and Debugging

Build repeatable eval datasets, test retrieval and tool behavior, monitor quality and cost, and debug production failures.

Build an Evaluation Dataset

An evaluation dataset captures representative inputs, expected properties, and edge cases. It should include ordinary cases, ambiguous cases, adversarial inputs, empty or malformed data, and cases where the correct behavior is to abstain. Keep evaluation examples versioned and separate from training or prompt examples where possible to avoid overfitting.

Evaluate the whole pipeline: retrieval recall, citation support, structured-output validity, tool argument correctness, authorization decisions, latency, and user outcomes. A single “good answer” score hides important failure modes. Use human review for nuanced judgments and calibrate automated graders against human ratings.

Evaluation case shape

{
  "id": "returns-ambiguous-03",
  "input": "Can I return this if I opened it?",
  "expectedBehavior": "Ask for product category if policy differs",
  "mustNot": ["Invent a universal return rule"],
  "tags": ["ambiguity", "abstention"],
  "policyVersion": "returns-v4"
}

💡 The expected behavior is expressed as criteria, not necessarily a single exact response. Include the relevant policy version and make evaluation cases stable across runs.

Key Takeaways

Version representative evaluation cases.
Measure component-level and end-to-end quality.
Calibrate automated graders against human review.

Knowledge Check

Why include cases where the correct answer is to abstain?

Unit, Integration, and Regression Testing

Unit-test deterministic components such as prompt assembly, schema validation, authorization, chunking, and tool execution. Mock provider calls in unit tests to avoid cost and nondeterminism. Integration tests can use a small number of live calls or provider sandboxes, while regression evals compare outputs across prompt, model, or retrieval changes.

Do not assert exact generated prose unless the output is intentionally deterministic. Assert required facts, schema validity, citations, forbidden behavior, and action boundaries. Add tests for timeouts, rate limits, partial streams, tool loops, retries, and cancellation.

Test a validation boundary

import { describe, it, expect } from "vitest";

describe("TicketSchema", () => {
  it("rejects an unknown category", () => {
    const result = TicketSchema.safeParse({
      category: "urgent-ish",
      priority: "high",
      summary: "Cannot sign in"
    });
    expect(result.success).toBe(false);
  });
});

💡 This test verifies a deterministic contract without calling a model. Add similar tests for authorization failures, invalid tool arguments, and unsupported output states.

Key Takeaways

Mock model calls in unit tests.
Test properties and safety constraints rather than exact prose.
Keep a regression suite for prompt/model/retrieval changes.

Knowledge Check

Which AI component is a strong candidate for ordinary deterministic unit tests?

Tracing, Metrics, Cost, and Production Debugging

Record a trace across the user request, retrieval, model call, tool execution, and final response. Useful metrics include request volume, success/error rate, time to first token, total latency, token usage, estimated cost, retrieval scores, tool call counts, and user feedback. Use correlation IDs to connect events without logging unnecessary raw content.

Monitor by feature and tenant so a costly or failing workflow can be isolated. Sample or redact content logs, restrict access to traces, and define retention. Model upgrades can change quality and tool behavior, so monitor drift and rerun evaluations before broad rollout.

Structured trace record

{
  "traceId": "trace_abc",
  "feature": "knowledge-assistant",
  "model": "configured-model",
  "retrievalCount": 6,
  "toolCalls": 1,
  "latencyMs": 1840,
  "inputTokens": 1200,
  "outputTokens": 220,
  "status": "success"
}

💡 Use provider usage fields when available and label cost as estimated if pricing is calculated separately. Avoid recording raw prompts or outputs by default if they may contain sensitive data.

Key Takeaways

Trace across retrieval, model, tools, and output validation.
Track cost and latency by feature and tenant.
Redact content and control trace access and retention.

Knowledge Check

Which is a useful production metric for streaming chat?

Build an Evaluation Dataset

Unit, Integration, and Regression Testing

Tracing, Metrics, Cost, and Production Debugging

STAY UPDATED WITH BYTES!