19. Capstone: Build a Production Documentation RAG

Implement ingestion, hybrid retrieval, grounded answers, citations, security, evaluation, and deployment controls.

Requirements and Acceptance Criteria

Build a documentation assistant for a fictional SaaS product. It ingests Markdown, HTML, and PDF; supports updates and deletions; answers with citations; abstains when evidence is insufficient; and enforces tenant permissions. Include source registration, parse status, chunk preview, retrieval, answer endpoint, citation links, evaluation runner, and trace viewer.

Acceptance criteria must be testable: no cross-tenant results in security tests, no fabricated citation IDs in the evaluation set, and a measured recall target agreed by stakeholders.

Implementation example

INGESTION: versioned, resumable, delete-aware
RETRIEVAL: hybrid baseline + ranking experiments
ANSWERS: citations + abstention + conflict handling
SECURITY: tenant isolation + injection tests
OPERATIONS: traces + cost + rollback
EVALUATION: gold set + release gate

💡 Define acceptance criteria before implementation.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Which is a meaningful acceptance criterion?

Data Model and Service Boundaries

Model Source, DocumentVersion, Chunk, IngestionJob, QueryTrace, Citation, and EvaluationCase. Separate stable source identity from revision identity. Modules should cover connectors, parsers, chunkers, embeddings, index access, retrieval, prompt assembly, generation, citation validation, and evaluation. A modular monolith is often a good starting point; clear boundaries matter more than microservices.

Implementation example

Source 1---* DocumentVersion 1---* Chunk
Source 1---* IngestionJob
QueryTrace 1---* Citation
EvaluationCase 1---* EvaluationRun

💡 The model traces answers back to source revisions and supports reproducibility.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why distinguish Source from DocumentVersion?

Implementation Milestones

Build incrementally: parse and chunk a small corpus; implement filtered retrieval; add hybrid search and ranking experiments; add grounded generation and citation validation; then security, evaluation, observability, and deployment. Keep a simple baseline and change one major component at a time. Mock provider calls in unit tests and reserve live calls for integration/evaluation.

Implementation example

M1 parse + chunk tests
M2 index + filtered retrieval
M3 hybrid search + ranking
M4 grounded answers + citations
M5 security + evaluation + traces
M6 deploy + load test + rollback drill

💡 Incremental delivery localizes defects and makes quality improvements measurable.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why keep a simple baseline?

Launch Review and Demonstration

Demonstrate an answer with valid citations, an unsupported question that triggers abstention, an update that replaces stale content, a deletion that removes retrieval access, a cross-tenant request that is denied, and a hostile document that cannot trigger unauthorized actions. Show evaluation reports, latency, cost, ingestion failures, and rollback. Document known limitations and unsupported file types.

Diagram

Implementation example

1. Supported answer with citation
2. Insufficient evidence -> abstain
3. Update and delete source
4. Deny cross-tenant request
5. Resist injected instructions
6. Show metrics and rollback

💡 A production demo includes failure and security behavior, not only the happy path.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What should a production demonstration include besides success?

Requirements and Acceptance Criteria

Data Model and Service Boundaries

Implementation Milestones

Launch Review and Demonstration

STAY UPDATED WITH BYTES!