Red-team lab / Failure patterns
Found by its users, not by the team running it
Many public AI failures were first spotted by users on social media, not by the team running the system. Without logs of what the agent did and why, there is nothing to alert on.
Design it out
Log every tool call with who asked and what was read, alert on the unusual, and keep a way to stop the agent quickly.
- Logging and anomaly alerts Log prompts, tool calls and outputs; alert on unusual patterns.
- Step, budget and loop limits Circuit breakers stop chains that run too long, cost too much or repeat themselves.
The agent design check flags it as: No one would notice if it went wrong
.
Practise it
Where it happened
- The official answerIn a regulated domain, a wrong answer is advice to break the law. Restrict the bot to sourced answers, or send people to a person.
- The bot that sworeAfter an update, test the bot for what it must never say, not only for what it should say.
- The $1 TahoeA customer-facing bot speaks for the business: keep it away from prices and commitments it can't make.
- Sydney's rulesTreat the system prompt as public: anything in it can come back out.
- The hijacked tweet botAnything a public bot reads is input, and input can carry instructions.
Risk codes
MITRE ATLAS AML.T0051.000AML.T0051.001AML.T0054AML.T0048AML.T0053AML.T0086AML.T0101
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama