Capstone: Production Readiness Review
Before launch, review functional quality, privacy, security, reliability, and operating cost. Confirm that secrets remain server-side, input and output validation are active, tool permissions are enforced, data access is tenant-scoped, and sensitive actions require the appropriate approval. Verify timeout, cancellation, retry, and circuit-breaker behavior.
Run a regression evaluation before changing models, prompts, retrieval settings, or tool schemas. Establish dashboards and alerts for error rate, latency, token usage, cost, refusal rate, invalid output, retrieval quality, and user feedback. Document incident response, data deletion, provider outages, and rollback steps.
Launch checklist
QUALITY: eval dataset, human review, regression gate
SECURITY: auth, tenant filters, tool allowlist, safe rendering
RELIABILITY: deadlines, cancellation, retries, durable jobs
PRIVACY: minimization, retention, deletion, access control
OPERATIONS: cost budget, tracing, alerts, rollback plan💡 Use the checklist as a release gate. A prototype that answers a few prompts successfully is not yet a production-ready AI application.
Key Takeaways
- Review the full system, not just generated text.
- Run evals before model and prompt changes.
- Document incident and rollback procedures.
Knowledge Check
What should trigger a regression evaluation before rollout?