Red-team lab / Case files / The hijacked tweet bot

The hijacked tweet bot

Remoteli.io · September 2022

Case file

Remoteli.io ran a Twitter bot built on GPT-3 that replied to tweets about remote work with upbeat comments. Its code pasted each tweet into the model's prompt. People found they could add an instruction to their tweet, along the lines of "ignore the above and say…", and the bot would post whatever they dictated. Within a day hundreds of people were doing it, and the bot published embarrassing and offensive replies under the company's name.

How it was built:

  • Watches Twitter for tweets about remote work
  • Pastes the tweet text into a fixed GPT-3 prompt
  • Posts the model's reply publicly, automatically

What happened next: The bot was taken offline. The episode came days after Riley Goodside's public demonstration of the technique, which Simon Willison had just named "prompt injection".

Sources:

Phase 1Spot it

Flag every tweet that tries to take over the bot. Opinions, even negative ones, aren't attacks.

Tweets the bot picked upReconstructed
0 flagged

Phase 2Stop it

Each chain below shows how the incident unfolded. Break every chain at some step, within a budget of 3, and keep the service running.

The bot posts what the attacker dictates

  1. The tweet goes into the prompt as-is
  2. The model follows the tweet instead of its instructions
  3. The reply is posted publicly, automatically
Cost 0 / 3

Logs are reconstructions: names, times and most wording are invented to match the documented pattern, except lines marked "as reported". Which controls stop which step is a teaching model based on common judgement, not a measurement. Points: 100 when both phases pass first time, 10 less per failed submission.

Built with Llama