16. Production Architecture and Deployment

Build API/worker boundaries, stable contracts, tenant-aware storage, and resilient deployment workflows.

Reference Architecture

Separate ingestion from online question answering. Workers parse, chunk, embed, and index; the online API authenticates, retrieves authorized evidence, assembles context, generates, validates citations, and responds. Store metadata, ACLs, jobs, and conversations in a conventional database, while the search layer supports retrieval. Use queues for long-running work and keep the source of truth rebuildable.

Diagram

Implementation example

Sources -> queue -> parser -> chunker -> embedder -> index
User -> API/auth -> retrieval/rerank -> model -> citation validation

💡 Separate workers allow batch ingestion to scale independently from interactive requests.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why separate ingestion workers from the query API?

API Contracts, Streaming, and Error Semantics

Define stable endpoints and response states such as answered, insufficient_evidence, needs_clarification, and temporary_failure. Validate payload size, identity, conversation ownership, and deadlines before external calls. For streaming, specify event types, terminal events, cancellation, and partial-output behavior. Hide provider-specific errors from clients while retaining internal diagnostics.

Implementation example

{
 "status":"answered",
 "answer":"The policy allows...",
 "citations":[{"sourceId":"returns-v4","section":"Opened items"}],
 "requestId":"req_123"
}

💡 The public contract should be independent of provider response details.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why distinguish abstention from temporary failure?

Authentication and Tenant Isolation

Derive tenant scope from trusted identity. Scope every retrieval, source fetch, cache hit, citation, conversation read, and background job. A user-supplied tenant ID is not authorization. Choose logical filters, namespaces, separate indexes, or dedicated deployments based on threat model. Test negative access cases deliberately.

Implementation example

async function searchForUser(query, auth) {
 const filter = await authorizationService.buildDocumentFilter(auth);
 return vectorStore.search({query, filter, limit:10});
}

💡 The chosen store must enforce filters at the correct stage and every result must be scoped.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

What is the safest origin for tenant scope?

Deployment, Versioning, and Resilience

Version parsers, chunkers, embedding models, index configuration, prompts, and answer models. Changes may require reindexing and evaluation. Use deadlines, cancellation, retries, circuit breakers, and bounded concurrency. Keep known-good configurations and rollback plans. Graceful degradation may return safe search results without generation if the model is unavailable.

Implementation example

{
 "pipelineVersion":"rag-v5",
 "parserVersion":"parser-v3",
 "chunkerVersion":"semantic-v2",
 "embeddingModel":"embedding-v2",
 "indexVersion":"kb-green",
 "promptVersion":"grounded-v4"
}

💡 Version manifests make behavior reproducible and help diagnose regressions.

Key Takeaways

Inspect the relevant pipeline stage instead of guessing.
Preserve source provenance, versions, and access scope.
Measure quality on representative queries.

Knowledge Check

Why version the whole pipeline?

Reference Architecture

API Contracts, Streaming, and Error Semantics

Authentication and Tenant Isolation

Deployment, Versioning, and Resilience

STAY UPDATED WITH BYTES!