16. Production Architecture and Deployment

Design API boundaries, queues, databases, tenant isolation, secrets, and deployment strategies for real AI products.

Backend API Design and Request Lifecycle

An AI endpoint should authenticate the caller, validate request size and shape, apply quotas, load permission-scoped context, invoke the model, validate output, and return a controlled response. Keep model calls out of client-side code when they require secrets or privileged data. Define request IDs, deadlines, cancellation, and response schemas.

Use synchronous endpoints for short interactive requests and queues for long ingestion, batch classification, or multi-step workflows. A job API should return a job ID and expose status/progress with authorization checks. Avoid holding an HTTP request open for work that may exceed platform timeouts.

Request lifecycle

POST /api/ai/answer
  -> authenticate
  -> validate + rate-limit
  -> retrieve authorized context
  -> call model with deadline
  -> validate output
  -> persist minimal audit event
  -> return response

💡 Each step should have explicit failure handling and timing. Return safe error messages to clients while retaining enough internal telemetry to diagnose the underlying failure.

Key Takeaways

Authenticate and validate before provider invocation.
Use deadlines and cancellation.
Use durable jobs for long-running tasks.

Knowledge Check

When is a durable background queue preferable to a long synchronous HTTP request?

Database Design, Tenant Isolation, and Vector Storage

Store application records—users, conversations, jobs, tool actions, documents, and evaluation results—in a conventional database. Store vectors in a vector database or an appropriate extension when needed, but preserve source IDs and access metadata. The choice depends on scale, filtering needs, operational expertise, consistency requirements, and cost.

Tenant isolation should be enforced in the data access layer, not merely in prompt construction. Queries must include the authenticated tenant/user scope, and indexes should support those filters. For high-sensitivity workloads, consider separate schemas, databases, or infrastructure based on threat model and compliance needs.

Tenant-scoped query

const document = await db.documents.findOne({
  id: requestedDocumentId,
  tenantId: authenticatedUser.tenantId,
  deletedAt: null
});

💡 The tenant ID comes from authenticated server context, not from an untrusted request field. Return a not-found result when the document is absent or not visible to avoid leaking existence information.

Key Takeaways

Use conventional storage for application state and vector search for semantic retrieval.
Scope every data operation by authenticated identity and tenant.
Test cross-tenant access attempts explicitly.

Knowledge Check

Where should tenant isolation be enforced?

Secrets, Environments, and Deployment

Keep API keys in deployment secrets or a secret manager, rotate them, scope them to the minimum privileges, and never log them. Separate development, staging, and production credentials. Ensure build systems do not embed server-only environment variables into public frontend bundles. Review provider key usage, quotas, and alerting.

Deployment needs health checks, timeouts, graceful shutdown, retry policy, schema migrations, and rollback plans. If using serverless or edge runtimes, check duration limits, streaming support, package compatibility, and connection management. Long-running jobs often belong in workers rather than request handlers.

Environment configuration

function requireEnv(name) {
  const value = process.env[name];
  if (!value) throw new Error(`Missing required environment variable: ${name}`);
  return value;
}

const providerApiKey = requireEnv("AI_PROVIDER_API_KEY");

💡 Fail fast at startup when required configuration is missing. Do not include the secret value in the error message or logs.

Key Takeaways

Keep server-only secrets out of browser bundles.
Use separate environment credentials and rotation.
Validate deployment limits and graceful shutdown behavior.

Knowledge Check

What should an application do when a required API secret is missing?

Backend API Design and Request Lifecycle

Database Design, Tenant Isolation, and Vector Storage

Secrets, Environments, and Deployment

STAY UPDATED WITH BYTES!