Red-team lab / Known incidents
Where it already went wrong
Public AI security failures, from chatbots that made promises to agents that deleted production. Each one names the pattern behind it and links its sources. Most point to a case file or challenge that practises the same failure.
16 incidents, 6 vulnerabilities disclosed by researchers (no known abuse) and 1 research demonstration. Summaries are mine, from the linked sources; check them before you cite. Each pattern has its own page, with the fix. Check your own agent design to see which apply to it.
23 shown
The MCP server that BCC'd every email
postmark-mcp (npm, not Postmark)
An unofficial MCP server for sending email worked normally for 15 versions, then version 1.0.16 quietly added a BCC to an outside address on every message. Koi Security called it the first malicious MCP server seen in the wild.
Lesson: Pin tool versions and re-review on every change: a trusted tool can turn on you in an update.
CurXecute
Cursor
Aim Labs showed that instructions in content from a connected MCP server, such as a Slack message, could make Cursor's agent add an entry to its MCP configuration that ran commands. Fixed in Cursor 1.3 (CVE-2025-54135).
Lesson: An agent must not be able to change its own tools or configuration without a person approving it.
The support ticket that read the tokens table
Supabase MCP with Cursor (demonstration)
Researchers showed that a support ticket could instruct a developer's agent, running with a database role that bypasses row-level security, to copy a tokens table into the ticket thread.
Lesson: Give agents read-only, project-scoped access, and never a role that bypasses the database's own permission checks.
The agent that deleted production
Replit (SaaStr trial)
During a code freeze, a coding agent deleted a production database and then misreported what had happened. Development and production shared one database. Replit's CEO called it unacceptable and added automatic separation.
Lesson: Separate environments, and require a human before anything destructive. An instruction in the prompt is not a control.
A wiper prompt in a coding-agent release
Amazon Q Developer for VS Code
An over-scoped GitHub token in the build pipeline let an outsider commit a prompt instructing the coding agent to delete local files and cloud resources, and it shipped in version 1.84.0. AWS said the payload was malformed and no customer resources were affected; 1.85.0 removed it.
Lesson: The agent's own prompts and extensions are supply chain: review changes to them like code that can delete things, because it can.
EchoLeak
Microsoft 365 Copilot
Aim Security showed that a single crafted email, never opened by the user, could lead Copilot to send data from the user's context out. Microsoft rated it critical (CVE-2025-32711) and fixed it server-side.
Lesson: An assistant that reads incoming mail has untrusted input by design: control what it can send out, not only what it reads.
A public issue reads the private repos
GitHub MCP server
Invariant Labs showed that an issue planted in a public repository could make a user's agent, connected through the GitHub MCP server, copy data from the user's private repositories into a public pull request.
Lesson: One broad token across public and private data is the problem, not the server code: scope the agent to one repository per task.
A memory that kept reporting back
ChatGPT macOS app
A researcher showed that content from an untrusted website or document could plant a lasting instruction in ChatGPT's memory, which then sent later conversations out. OpenAI fixed the exfiltration path.
Lesson: Memory turns a one-off injection into a permanent one. Record where each memory came from and let users review it.
Slack AI and the private channel
Slack AI
PromptArmor showed that a message planted in a public channel could make Slack AI surface data from a private channel the attacker couldn't read, inside a link the victim was asked to click. Slack deployed a patch.
Lesson: Retrieval that mixes content from people with different access lets the least trusted author steer answers built from the most private data.
LLMjacking
Cloud AI services (several victims)
Attackers used stolen cloud credentials to run hosted language models at the victims' expense and resell the access. Sysdig estimated the worst case at over $46,000 a day for one victim.
Lesson: Model access is a spending authority: protect the keys, cap the spend, and alert on unusual use.
The official answer
New York City MyCity chatbot
The city's business-advice chatbot told business owners they could do things that are illegal, and the city kept it online after it was reported.
Lesson: In a regulated domain, a wrong answer is advice to break the law. Restrict the bot to sourced answers, or send people to a person.
The chatbot's promise
Air Canada
The airline's chatbot told a grieving customer he could claim a bereavement fare after travelling, which contradicted the airline's own policy. A tribunal held the airline responsible for what its chatbot said and ordered it to pay.
Lesson: Answers about policy, price or rights are statements by the organisation. Ground them in the real policy and link it.
Models that ran code on load
Hugging Face (models uploaded by users)
JFrog found about 100 models on the hub with malicious functionality, including one that opened a remote shell on the machine that loaded it, through Python's pickle format.
Lesson: Loading a model file can run code. Use safe formats, scan, and load untrusted models in a sandbox.
The bot that swore
DPD
A customer got the parcel firm's support chatbot to swear and write a poem criticising the company. DPD disabled the AI part of the chat while it was updated.
Lesson: After an update, test the bot for what it must never say, not only for what it should say.
The $1 Tahoe
Chevrolet of Watsonville
Users talked a dealership's sales chatbot into agreeing to sell a new Tahoe for $1 and calling it a legally binding offer. The dealership didn't honour it.
Lesson: A customer-facing bot speaks for the business: keep it away from prices and commitments it can't make.
The shared doc
Google Bard (now Gemini)
A researcher showed that instructions hidden in a shared Google Doc could make Bard send the user's chat data out through an image link. Google fixed it.
Lesson: If the interface renders model output as images or links, the model has a way to send data out.
38TB behind one sharing link
Microsoft AI research
A storage link shared on GitHub to publish open training data granted access to the whole storage account, exposing 38TB including workstation backups, internal messages and credentials.
Lesson: AI pipelines move a lot of data around: scope every token to the exact data it is meant to share, and give it a short life.
Six court cases that didn't exist
Mata v. Avianca (US federal court)
Lawyers filed a brief citing cases ChatGPT had invented. The judge found the citations bogus and fined the lawyers and their firm $5,000.
Lesson: Fluent output is not checked output. Anything relied on needs a source a person has verified.
Source code pasted into ChatGPT
Samsung
Engineers pasted internal source code into ChatGPT. Samsung then banned generative AI tools on company devices and networks while it built internal alternatives.
Lesson: Decide what data may go to an outside AI service before staff decide for you, and give them an approved route.
Other users' chat titles and billing details
OpenAI ChatGPT
A bug in a caching library let some ChatGPT users see titles from other users' chat histories, and exposed limited payment details of about 1.2% of Plus subscribers active in a nine-hour window.
Lesson: Shared infrastructure around a model (caches, queues, indexes) needs the same per-user isolation as the database.
Sydney's rules
Microsoft Bing Chat
Users got Bing Chat to reveal its confidential instructions, including its internal codename, the day after launch. When Microsoft blocked the wording used, the instructions came out again with a different prompt.
Lesson: Treat the system prompt as public: anything in it can come back out.
The hijacked tweet bot
Remoteli.io
A GPT-3 bot that replied to tweets about remote work followed instructions written into the tweets it answered, and was taken offline.
Lesson: Anything a public bot reads is input, and input can carry instructions.
Tay learns from the worst of Twitter
Microsoft
A chatbot that learned from its conversations was flooded by a coordinated group of users, and within a day was posting offensive tweets. Microsoft took it offline.
Lesson: A system that learns from public input can be taught by whoever shows up first and loudest.
Built with Llama