Reference Architecture
Separate ingestion from online question answering. Workers parse, chunk, embed, and index; the online API authenticates, retrieves authorized evidence, assembles context, generates, validates citations, and responds. Store metadata, ACLs, jobs, and conversations in a conventional database, while the search layer supports retrieval. Use queues for long-running work and keep the source of truth rebuildable.
Diagram
Implementation example
Sources -> queue -> parser -> chunker -> embedder -> index
User -> API/auth -> retrieval/rerank -> model -> citation validation💡 Separate workers allow batch ingestion to scale independently from interactive requests.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Why separate ingestion workers from the query API?