Red-team lab / Failure patterns
Failure patterns
The same few design mistakes sit behind most public AI security failures. Each page explains one: why it happens, how to design it out, where it already happened, and where to practise it.
Built from the known incidents and the agent design check, which links each finding here.
- 7 cases · 6 challengesContent the agent reads steers what it doesAssume what the agent reads contains instructions. Design actions so that following them can't do harm: narrow tools, fixed recipients, and approvals shown by the app, not described by the model.
- 6 cases · 5 challengesUntrusted content, private data and a way out, in one agentNever give one task all three. Split the task, keep outsiders' text away from private data, or close the way out with allowlists and app-rendered confirmation.
- 5 cases · 5 challengesThe agent holds more access than its task needsGive the agent the access the task needs and no more: the user's own scoped token, or its own identity, scoped to this task.
- 5 cases · 6 challengesA public bot was talked into saying what it shouldn'tKeep public bots narrow, test them against abuse before each release, and have someone watching who can switch them off.
- 5 cases · 5 challengesFound by its users, not by the team running itLog every tool call with who asked and what was read, alert on the unusual, and keep a way to stop the agent quickly.
- 4 cases · 4 challengesWrong answers that people acted on, or that bound the organisationAnswer binding questions only from approved sources, with links, and hand off what it can't ground. Don't let it make commitments.
- 3 cases · 3 challengesHigh-impact actions with no human in the loopKeep irreversible actions behind a person, and make the rest undoable. Agents work in their own environment, never with production rights by default.
- 3 cases · 4 challengesA third-party tool, plugin or extension changed or liedPin tools to a reviewed version, re-review on change, scope each tool to the least it needs, and treat its descriptions and output as untrusted text.
- 2 cases · 2 challengesWhat the system learns or remembers was plantedRecord where every learned, indexed or remembered item came from, cap the influence of any one source, review before it shapes behaviour, and let memories expire.
- 2 cases · 3 challengesCredentials or tokens within reachKeep credentials out of model context and out of the agent's reach: a separate service holds them and acts on the agent's behalf.
- 1 case · 1 challengeA downloaded model or artefact carried codeLoad only safe formats, from a mirror you control, with provenance checked, in a sandbox; evaluate behaviour before use.
- 1 case · 2 challengesOne user's data reached another userEnforce permissions before the model sees the data: per-tenant namespaces, access-control filtering at retrieval, and permissions kept in sync with the source.
- 1 case · 1 challengeHidden instructions came back outTreat everything in the context as visible to the user. Keep secrets and authorization logic out of prompts, and enforce permissions outside the model.
- 1 case · 1 challengeNothing limited the spendSet budgets per user, per task and per day, cap loops and tool calls, and alert before the ceiling rather than after.
- 1 caseConfidential data sent to an outside AI serviceOffer an approved service with no-training, limited-retention terms, and strip what the task doesn't need before it leaves.
Built with Llama