Why Threat Modelling Matters for Software and AI Systems

Threat modelling is one of the most useful security activities I can do before writing code. It brings developers, product people, operations, and security into the same conversation: what are we building, what could go wrong, and what are we going to do about it?

The output is not a perfect diagram or a large document. It is a shared understanding of risk and a short list of actions the team can own.

Start with four questions

I use the four-question structure popularised by the Threat Modeling Manifesto and OWASP:

1. What are we working on?
2. What can go wrong?
3. What are we going to do about it?
4. Did we do a good enough job?

That is enough to run a useful first session. A framework such as STRIDE can help the team think, but the framework should support the conversation rather than dominate it.

A session engineers will tolerate

I keep the first session to about an hour and invite people who understand the product, architecture, operations, and security. Before the meeting, I prepare a simple system and data-flow diagram.

1. Agree the scope

Pick one feature or flow. “Threat model the platform” is too broad. “Register a vehicle and let its owner view it” is workable.

List the important assets and trust boundaries:

• personal or commercially sensitive data;
• credentials and tokens;
• actions with financial or operational impact;
• transitions between browsers, APIs, services, queues, models, and data stores;
• third parties the team does not control.

2. Describe failure in plain language

I ask everyone to complete a sentence:

Someone or something can by , which would cause ___.

Plain threat statements are easier to discuss than isolated security terms. “A signed-in user can change the vehicle ID in an API request and read another customer’s record” gives the team something concrete to test and fix.

3. Choose proportionate controls

For each important threat, record prevention, detection, and recovery where they make sense. A control needs an owner and a way to verify it.

“Use access control” is not an action. “The API checks the authenticated customer ID against the vehicle owner for every read, covered by an integration test” is.

4. Decide what remains

Not every risk needs immediate engineering work. The team can mitigate, avoid, transfer, or consciously accept a risk. Record the decision and the person accountable for it.

A small example

Imagine a vehicle registration service with a web application, an API, an identity provider, and a database.

The point is not to name every possible attack. It is to identify the paths with meaningful impact and make the design better before they become incidents.

What changes for AI systems

An AI feature adds new components and less deterministic behaviour, but it still sits inside an ordinary software system. I threat model the complete path, not only the model.

I pay particular attention to:

• Prompt injection: untrusted content tries to change instructions or trigger tools.
• Excessive agency: the model can take a high-impact action without enough checks.
• Data leakage: prompts, retrieval results, traces, or outputs expose information across users or tenants.
• Retrieval poisoning: indexed content is manipulated so the system retrieves harmful instructions or false information.
• Insecure output handling: generated text is passed to SQL, a shell, HTML, or another system without validation.
• Model and tool supply chain: external models, MCP servers, plugins, and data sources change outside the team’s release process.

The controls are familiar engineering controls applied carefully: isolate untrusted content, constrain tools, validate outputs, enforce authorisation outside the prompt, require approval for risky writes, and monitor unusual behaviour.

Telling the model to “ignore malicious instructions” is not a sufficient control.

Keep the model alive

A threat model goes stale when the system changes. I revisit it when the team adds a new data source, tool, external integration, permission model, or high-impact action. Incidents and near misses are also useful triggers.

The threat list should connect to normal work:

• mitigations in the backlog;
• security acceptance criteria;
• automated tests;
• monitoring and alerts;
• penetration-test scope;
• architecture decisions.

If nobody uses the document after the workshop, the process has failed.

Common mistakes

I avoid these patterns:

• trying to model the whole organisation in one meeting;
• inviting only security specialists;
• producing a diagram without decisions;
• treating every threat as equally urgent;
• relying on a tool to replace discussion;
• running the exercise once and never updating it.

Threat modelling works best as a small, repeated design habit. It does not remove risk, but it makes risk visible early enough for the team to make a deliberate choice.

References

• OWASP Threat Modeling
• Threat Modeling Manifesto
• Threat modelling a generative AI workload