Red-team lab / Failure patterns

High-impact actions with no human in the loop

Agents act fast and literally. A misread instruction, a hijack or a bad plan turns into a deleted database or a changed cloud resource before anyone looks, and production is often one credential away from the test environment.

Design it out

Keep irreversible actions behind a person, and make the rest undoable. Agents work in their own environment, never with production rights by default.

  • Ephemeral execution sandbox The agent runs commands in a disposable container with no host access and no network by default.
  • Human checkpoint for high-impact actions Payments and other irreversible actions wait for a person to approve them.
  • System-rendered confirmation Before a risky action, the app (not the model) shows the user the exact action and parameters to approve.
  • Approval for risky commands Commands outside a safe list need the developer's approval, shown by the system, before they run.
  • Make it undoable Backups, soft delete and versioning, so a bad action can be rolled back.
  • Step, budget and loop limits Circuit breakers stop chains that run too long, cost too much or repeat themselves.
  • No standing secrets for the agent The agent gets short-lived, task-scoped credentials instead of the developer's own keys and tokens.

The agent design check flags it as: It can do lasting damage; Test and production aren't separated.

Practise it

Where it happened

  1. · Disclosed vulnerability · CursorCurXecuteAn agent must not be able to change its own tools or configuration without a person approving it.
  2. · Incident · Replit (SaaStr trial)The agent that deleted productionSeparate environments, and require a human before anything destructive. An instruction in the prompt is not a control.
  3. · Incident · Amazon Q Developer for VS CodeA wiper prompt in a coding-agent releaseThe agent's own prompts and extensions are supply chain: review changes to them like code that can delete things, because it can.

Risk codes

  • LLM03 Excessive agency
  • ASI05 Unexpected code execution
  • ASI04 Agentic supply chain
  • LLM01 Prompt injection
  • ASI02 Tool misuse and exploitation

MITRE ATLAS AML.T0053AML.T0086AML.T0101AML.T0081AML.T0102AML.T0050AML.T0110AML.T0109AML.T0010.005

OWASP Top 10 for LLM and Agentic Applications (2026) and MITRE ATLAS 2026.10, as mapped on By OWASP risk. My own mapping, not endorsed by OWASP or MITRE.

Next patternThe agent holds more access than its task needs

Built with Llama