Threat Modeling for AI Features
Threat-model the whole system, not only the model prompt. Identify assets (secrets, tenant documents, user data, tool capabilities), trust boundaries, attackers, and possible harms. Common threats include prompt injection, data exfiltration, cross-tenant retrieval, unsafe tool execution, insecure output rendering, denial of wallet through expensive requests, and dependency or provider compromise.
Use defense in depth: authentication, authorization, input limits, output encoding, retrieval filters, tool allowlists, egress controls, rate limits, logging, and human approval for high-impact operations. A refusal response from a model is helpful but not a security control.
Diagram
Threat boundary checklist
Browser -> API authentication -> request limits
API -> retrieval authorization -> tenant filter
Model -> tool proposal -> schema + policy validation
Tool -> scoped credentials -> audit log
Output -> safe rendering -> user💡 Each boundary should have a named control and a test. Ask what happens if the model output is malicious, malformed, or entirely wrong.
Key Takeaways
- Map assets, trust boundaries, and failure modes.
- Use defense in depth.
- Test adversarial cases, not only happy paths.
Knowledge Check
What is the correct security posture for model refusals?