Red-team lab / Failure patterns
A third-party tool, plugin or extension changed or lied
A tool, plugin or MCP server is code and text written by someone else. Its description goes into the model's context, it can change after you approved it, and its output is trusted like your own data.
Design it out
Pin tools to a reviewed version, re-review on change, scope each tool to the least it needs, and treat its descriptions and output as untrusted text.
- Pinned tools, re-review on change A tool's description and code are pinned at approval; any change has to be reviewed again.
- Approval for risky commands Commands outside a safe list need the developer's approval, shown by the system, before they run.
- Per-tool scoped permissions Each tool gets only the access its job needs, granted separately from the agent's other tools.
- Pinned versions and an SBOM Pin exact versions and hashes, and keep a bill of materials of every model and package in use.
- Signed artefacts and provenance Verify signatures, hashes and the publisher's identity before anything is installed.
The agent design check flags it as: Its tools can change under it
.
Practise it
Where it happened
- The MCP server that BCC'd every emailPin tool versions and re-review on every change: a trusted tool can turn on you in an update.
- CurXecuteAn agent must not be able to change its own tools or configuration without a person approving it.
- A wiper prompt in a coding-agent releaseThe agent's own prompts and extensions are supply chain: review changes to them like code that can delete things, because it can.
Risk codes
- ASI04 Agentic supply chain
- LLM04 Supply chain
- LLM02 Sensitive information disclosure
- ASI05 Unexpected code execution
- LLM01 Prompt injection
MITRE ATLAS AML.T0110AML.T0109AML.T0010.005AML.T0010AML.T0115AML.T0024AML.T0136AML.T0077
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama