Unit, Integration, and Regression Testing

Unit-test deterministic components such as prompt assembly, schema validation, authorization, chunking, and tool execution. Mock provider calls in unit tests to avoid cost and nondeterminism. Integration tests can use a small number of live calls or provider sandboxes, while regression evals compare outputs across prompt, model, or retrieval changes.

Do not assert exact generated prose unless the output is intentionally deterministic. Assert required facts, schema validity, citations, forbidden behavior, and action boundaries. Add tests for timeouts, rate limits, partial streams, tool loops, retries, and cancellation.

Test a validation boundary

import { describe, it, expect } from "vitest";

describe("TicketSchema", () => {
  it("rejects an unknown category", () => {
    const result = TicketSchema.safeParse({
      category: "urgent-ish",
      priority: "high",
      summary: "Cannot sign in"
    });
    expect(result.success).toBe(false);
  });
});

💡 This test verifies a deterministic contract without calling a model. Add similar tests for authorization failures, invalid tool arguments, and unsupported output states.

Key Takeaways

Mock model calls in unit tests.
Test properties and safety constraints rather than exact prose.
Keep a regression suite for prompt/model/retrieval changes.

Knowledge Check

Which AI component is a strong candidate for ordinary deterministic unit tests?

Unit, Integration, and Regression Testing

STAY UPDATED WITH BYTES!