Project 2: Grounded Documentation Assistant
Build a documentation assistant that ingests Markdown, HTML, and PDF documents, preserves source metadata, and answers questions with citations. Implement document updates and deletions, tenant-aware retrieval, chunking experiments, and a “not enough evidence” response. Create evaluation questions with expected source documents and unsupported questions that should trigger abstention.
Measure retrieval recall, citation support, answer correctness, latency, and cost. Test prompt injection in documents, cross-tenant retrieval attempts, stale content, and broken links. Add an admin view for ingestion status and reindexing errors.
Diagram
RAG capstone acceptance criteria
Required:
1. Source-aware ingestion and deletion
2. Permission-filtered retrieval
3. Grounded answers with validated citations
4. Abstention for insufficient evidence
5. Evaluation set and regression report
6. Cost/latency telemetry and failure recovery💡 The acceptance criteria cover product quality and operational reliability, not just whether the model can answer a sample question.
Key Takeaways
- Preserve provenance and permission filters.
- Test stale documents and deletion behavior.
- Evaluate citation support and abstention.
Knowledge Check
Which test is especially important for a multi-tenant documentation assistant?