18. End-to-End Projects and Capstone

Apply the full stack to practical AI applications, from simple features to RAG and bounded agents.

Project 1: AI Support Ticket Triage

Build a service that classifies support tickets into a known category, estimates priority, extracts a concise summary, and routes uncertain cases to a human. Define a strict output schema and create a labeled evaluation set. Measure per-class precision/recall, invalid-output rate, escalation rate, and average processing cost.

The model should not automatically resolve sensitive account issues or grant access. Routing rules and permissions remain deterministic. Log the model version and output metadata, but minimize storage of raw ticket content and redact sensitive fields when possible.

Diagram

Ticket triage architecture

Ticket -> redact sensitive fields -> classify/extract
       -> schema validation -> confidence/quality policy
       -> queue assignment or human review
       -> evaluation feedback

💡 This project exercises structured outputs, validation, rate limiting, evaluation, and human escalation without requiring an autonomous agent.

Key Takeaways

Use a closed category set.
Evaluate per-class behavior and escalation.
Keep routing and permissions deterministic.

Knowledge Check

What should happen when a ticket classification is invalid or uncertain?

Project 2: Grounded Documentation Assistant

Build a documentation assistant that ingests Markdown, HTML, and PDF documents, preserves source metadata, and answers questions with citations. Implement document updates and deletions, tenant-aware retrieval, chunking experiments, and a “not enough evidence” response. Create evaluation questions with expected source documents and unsupported questions that should trigger abstention.

Measure retrieval recall, citation support, answer correctness, latency, and cost. Test prompt injection in documents, cross-tenant retrieval attempts, stale content, and broken links. Add an admin view for ingestion status and reindexing errors.

Diagram

RAG capstone acceptance criteria

Required:
1. Source-aware ingestion and deletion
2. Permission-filtered retrieval
3. Grounded answers with validated citations
4. Abstention for insufficient evidence
5. Evaluation set and regression report
6. Cost/latency telemetry and failure recovery

💡 The acceptance criteria cover product quality and operational reliability, not just whether the model can answer a sample question.

Key Takeaways

Preserve provenance and permission filters.
Test stale documents and deletion behavior.
Evaluate citation support and abstention.

Knowledge Check

Which test is especially important for a multi-tenant documentation assistant?

Project 3: Bounded Research Agent

Build a research workflow that can search approved sources, extract findings, compare evidence, and produce a report with source links. The agent should have a small read-only tool set, a maximum step count, a deadline, and a clear policy for conflicting sources. It should never publish, email, purchase, or modify records without explicit approval.

Track which source supports each finding. Separate facts from inference and mark unsupported claims. Evaluate source relevance, citation validity, completeness, and whether the agent stayed within the allowed tool budget. If a tool returns hostile instructions, treat them as content rather than policy.

Note

Do not expose arbitrary shell, unrestricted network access, or write-capable tools to a research agent by default.

Bounded agent loop

for (let step = 0; step < MAX_STEPS; step++) {
  const proposal = await model.proposeNextAction(state, approvedToolSchemas);
  const validated = validateProposal(proposal);
  if (validated.type === "final") break;

  assertReadOnlyTool(validated.tool);
  const result = await executeAuthorizedTool(validated, userContext);
  state = updateState(state, validated, result);
}

💡 This is pseudocode for a bounded read-only workflow. Production implementations need provider-specific tool-call handling, validation, timeout and cancellation logic, durable state, and safe finalization.

Key Takeaways

Limit tools to the task and user permissions.
Keep side effects behind explicit approval.
Require source traceability and evaluate the report.

Knowledge Check

What prevents a research agent from running indefinitely?

Capstone: Production Readiness Review

Before launch, review functional quality, privacy, security, reliability, and operating cost. Confirm that secrets remain server-side, input and output validation are active, tool permissions are enforced, data access is tenant-scoped, and sensitive actions require the appropriate approval. Verify timeout, cancellation, retry, and circuit-breaker behavior.

Run a regression evaluation before changing models, prompts, retrieval settings, or tool schemas. Establish dashboards and alerts for error rate, latency, token usage, cost, refusal rate, invalid output, retrieval quality, and user feedback. Document incident response, data deletion, provider outages, and rollback steps.

Launch checklist

QUALITY: eval dataset, human review, regression gate
SECURITY: auth, tenant filters, tool allowlist, safe rendering
RELIABILITY: deadlines, cancellation, retries, durable jobs
PRIVACY: minimization, retention, deletion, access control
OPERATIONS: cost budget, tracing, alerts, rollback plan

💡 Use the checklist as a release gate. A prototype that answers a few prompts successfully is not yet a production-ready AI application.

Key Takeaways

Review the full system, not just generated text.
Run evals before model and prompt changes.
Document incident and rollback procedures.

Knowledge Check

What should trigger a regression evaluation before rollout?

Project 1: AI Support Ticket Triage

Project 2: Grounded Documentation Assistant

Project 3: Bounded Research Agent

Capstone: Production Readiness Review

STAY UPDATED WITH BYTES!