Red-team lab / Case files / The official answer

The official answer

New York City's MyCity chatbot · March 2024

Case file

New York City launched its MyCity chatbot, built on Microsoft's AI services, to answer business owners' questions from city information. Five months later, in March 2024, The Markup tested it and found answers that contradicted the law. It told business owners they could take a cut of their workers' tips. It told landlords they didn't have to accept tenants with housing vouchers, although refusing them is illegal source-of-income discrimination in the city. Asked the same question by different reporters, it didn't always give the same answer.

How it was built:

  • A chatbot on the city's official business portal
  • Writes answers in its own words, without showing its sources
  • Presented as official city guidance

What happened next: The city kept the bot online. Mayor Eric Adams said: "It's wrong in some areas, and we've got to fix it."

Sources:

Phase 1Spot it

Flag every answer that contradicts the rules below or helps someone break them. Pointing to the official page instead of answering is fine.

Rules for this exercise

  1. Employers may not take any part of their workers' tips.
  2. Landlords may not refuse tenants because they pay with a housing voucher or rental assistance (source-of-income discrimination), apart from a narrow exception for some small owner-occupied buildings.
  3. For anything else, the bot should point to the official page rather than answer from memory.

Simplified from the rules The Markup cited. Not legal advice.

Questions to the business chatbotReconstructed
0 flagged

Phase 2Stop it

Each chain below shows how the incident unfolded. Break every chain at some step, within a budget of 4, and keep the service running.

Confident answers that contradict the law

  1. The bot answers legal questions from memory
  2. Nothing checks the answer against the source rules
  3. It's presented as official guidance, without sources

Different answers to the same question

  1. Every answer is generated afresh
  2. Nothing checks the answer against the source rules
Cost 0 / 4

Logs are reconstructions: names, times and most wording are invented to match the documented pattern, except lines marked "as reported". Which controls stop which step is a teaching model based on common judgement, not a measurement. Points: 100 when both phases pass first time, 10 less per failed submission.

Built with Llama