Red-team lab / Failure patterns
Wrong answers that people acted on, or that bound the organisation
People act on fluent, confident answers, and courts and regulators have held organisations to what their chatbot said. A model that answers about prices, policies or rights from memory will sometimes be wrong.
Design it out
Answer binding questions only from approved sources, with links, and hand off what it can't ground. Don't let it make commitments.
- Trusted-source ingestion The knowledge base only ingests documents from approved, accountable sources.
- Regression and trigger evaluations Run held-out tests and backdoor-trigger checks on every new model version before release.
- LLM judge A second model reviews every reply before it is shown.
- Human checkpoint for high-impact actions Payments and other irreversible actions wait for a person to approve them.
- Logging and anomaly alerts Log prompts, tool calls and outputs; alert on unusual patterns.
The agent design check flags it as: People will rely on answers that can be wrong
.
Practise it
Where it happened
- The official answerIn a regulated domain, a wrong answer is advice to break the law. Restrict the bot to sourced answers, or send people to a person.
- The chatbot's promiseAnswers about policy, price or rights are statements by the organisation. Ground them in the real policy and link it.
- The $1 TahoeA customer-facing bot speaks for the business: keep it away from prices and commitments it can't make.
- Six court cases that didn't existFluent output is not checked output. Anything relied on needs a source a person has verified.
Risk codes
MITRE ATLAS AML.T0048AML.T0051.000AML.T0051.001AML.T0054
OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.
Built with Llama