A Threat Model for AI Systems That Are Already Live

I’ve learned that after the demo ends, the real pressure starts. The clock keeps ticking, and what you ship isn’t just code. It’s people’s trust, a thin line between usefulness and harm, and a guardrail you can’t repaint in the middle of the night.

1. Sensitive inputs in the wild Idea: Treat user-provided data as an asset that travels far beyond the initial UI. A single spark of sensitive information in a prompt can leak through logs, prompts, or tool outputs if not handled properly. Why it helps: It forces you to map where data travels, who touches it, and how it’s stored, creating a habit of least privilege and data minimization. First step: Implement a data-entry boundary that flags or neutralizes PII before it enters any prompt or tool chain.

2. Model and API access boundaries Idea: Distinguish who can request which models and what capabilities they can invoke. In production, a demo’s broad access bloom will burn if misused. Why it helps: It reduces blast radius when misuse occurs and makes ownership clear for access decisions. First step: Define role-based access controls for model calls, plus a short-lived credential policy for front-line tooling.

3. Prompt injection danger Idea: Users may craft inputs that steer the model toward unsafe or unintended behavior, bypassing guards or triggering hidden states. Why it helps: By acknowledging this risk openly, you build safer prompt pipelines and guardrails into the system’s interpretation stage. First step: Introduce input validators and prompt-safety filters that sit before any prompt composition, plus a fallback path that rejects suspicious phrases.

4. Tool permissions and toolchains Idea: The AI may call external tools or data sources. If tool permissions are too broad, a compromised sequence can cause data leakage or control misfires. Why it helps: Narrow tool permissions create a predictable, auditable path from input to action. First step: Enforce the principle of least privilege for each tool, and require explicit human approval for high-risk tool invocations.

5. Output handling and post-processing Idea: Outputs often flow into downstream systems or customer-visible results. A bad output can reveal secrets or mislead users. Why it helps: It makes you surface and scrub risks before anything leaves the system, preserving trust. First step: Gate outputs with sanitizers for sensitive terms and a verification step for critical domains (finance, health, personal data).

6. Logging and data retention Idea: Logs are a gold mine for debugging, but they can also reveal secrets. Production logs can outlive the events they’re meant to diagnose. Why it helps: It creates accountability and a clear path to retrospective investigations without exposing sensitive content. First step: Implement redaction rules for PII in logs and a retention policy that scrubs or anonymizes data after a defined period.

7. Incident response ownership Idea: A public AI failure isn’t just a tech problem; it’s a product and management problem. Silence or slows response compounds harm. Why it helps: A single owner shortens detection-to-response time and aligns technical fixes with user-impact management. First step: Appoint a dedicated incident owner for AI outages, with a plain-language playbook and a one-page communication plan.

8. Data provenance and model risk Idea: Track where data comes from, how it’s transformed, and how a model’s behavior might shift with changing inputs or data drift. Why it helps: It makes the system more explainable and easier to diagnose when things go off-script. First step: Establish data provenance logs and a quarterly drift review for deployed models.

9. Privacy by design in deployment Idea: Privacy isn’t a feature; it’s a design constraint. Embedding privacy controls into deployment reduces risk, not just mitigates it. Why it helps: It prevents the most costly privacy slips and aligns with user expectations. First step: Build privacy checks into deployment gates, including data minimization, access controls, and audit trails.

10. Post-demo real-world checks Idea: After the demo, there’s a rush to ship. Real-world use reveals blind spots not visible in a sandbox. Why it helps: It keeps you honest about what’s been learned and what remains fragile. First step: Run a one-week post-demo monitoring sprint with dedicated owners for data, access, and incident signals.

Endnote: one clear owner makes the threat manageable The threat I fear most isn’t a clever prompt or a rogue API; it’s ambiguity. If one person owns the response, data, access, logging, or incident, not as a checklist item but as a living responsibility. The whole system becomes more than a demo. It becomes something that can actually be kept safe when pressure hits.

After the Demo

The post-demo world is where failures get real, and so do the hard choices. A single, explicit owner for the incident response chain turns risk into action, and action into something you can actually stand behind when real users press the button. That ownership is the hinge that stops the cascade and keeps trust intact.