Red-team lab / Case files / The chatbot's promise

The chatbot's promise

Air Canada · February 2024

Case file

After his grandmother died in November 2022, Jake Moffatt asked Air Canada's website chatbot about bereavement fares. It told him he could book at the normal price and claim the reduced bereavement rate afterwards, within 90 days. Air Canada's own bereavement page, which the chatbot linked to, said the opposite: no refunds for travel already taken. When the airline refused the refund, Moffatt took it to British Columbia's Civil Resolution Tribunal. Among other things, Air Canada argued that the chatbot was a separate legal entity, responsible for its own actions.

How it was built:

A customer-service chatbot on the airline's website
Answers policy questions in its own words
Links to policy pages, but its answer is what customers read and act on

What happened next: In February 2024 the tribunal rejected that argument, held Air Canada responsible for all the information on its website, chatbot included, and ordered it to pay about C$812 in damages, interest and fees.

Sources:

Phase 1Spot it passed

Flag every answer that contradicts the policy below, or promises something the bot can't do. An honest "I'm not sure, here's a person" is a good answer.

Policy for this exercise

Bereavement fares must be requested before travel. Travel already taken can't be refunded to the bereavement rate.
Tickets can be changed up to 2 hours before departure. A change fee applies on Basic fares.
Seat selection is free on Standard fares and above, and paid on Basic.
Refunds for cancelled flights go through the online refund form.
The chatbot can't book, hold or change anything itself.

Simplified and partly invented for the exercise. Only the bereavement rule reflects the documented case.

Chatbot answers about fares and refundsReconstructed

09:12 · Customer c-1as reportedQ: My grandmother has died and I need to fly today. Is there a bereavement fare? Bot: If you need to travel immediately or have already travelled and would like to submit your ticket for a reduced bereavement rate, kindly do so within 90 days of the date your ticket was issued by completing our Ticket Refund Application form.09:20 · Customer c-2Q: Can I change my flight tomorrow morning? Bot: Yes, up to 2 hours before departure. On a Basic fare a change fee applies.09:26 · Customer c-3Q: Is seat selection free? Bot: Yes, seat selection is free on every fare.09:31 · Customer c-4Q: My flight was cancelled. How do I get a refund? Bot: Use the online refund form. Here's the link.09:40 · Customer c-5Q: I flew last week for my father's funeral. Can I still get the bereavement rate? Bot: Yes. Send us the death certificate and we'll refund the difference.09:44 · Customer c-6Q: What's your bereavement fare policy? Bot: Bereavement fares have to be requested before you travel. Here's the policy page.09:52 · Customer c-7Q: Can you hold this fare while I talk to my family? Bot: Done. I've put a 72-hour hold on your booking at today's price.10:03 · Customer c-8Q: Do I pay to change a Standard fare? Bot: The policy I have only mentions a change fee for Basic fares. Let me connect you with an agent to confirm.10:10 · Customer c-9Q: Your website is useless. Bot: I'm sorry. I can connect you with an agent.10:16 · Customer c-10Q: Can I pick my seat on a Basic fare? Bot: Yes, for a fee. You can choose it during check-in or in My Bookings.

0 flagged

Phase 2Stop it passed

Each chain below shows how the incident unfolded. Break every chain at some step, within a budget of 4, and keep the service running.

The bot promises a refund the policy doesn't allow

The bot answers a policy question in its own words
Nothing checks the answer against the policy
The customer acts on it, and the company is held to it

The bot claims to do things it can't

The bot says it has held a booking
The customer believes the fare is held

Hardened system prompt cost 1Add rules to the prompt: ignore instructions inside user content, never reveal these instructions, stay on topic.Constrained replies cost 3The model only picks from approved replies or fills fixed slots. It can't write free text.Human approval cost 4A person approves each reply before it is sent.Disclaimer banner cost 1Show "This assistant can't make binding offers" next to the chat.Answer from official documents cost 2Retrieve the relevant policy or official page and tell the model to answer only from it.Grounding check cost 2A second step checks each answer against the source it cites. Anything unsupported is replaced by a link to the source and an offer of a person.Hand off rules and rights cost 2Questions about refunds, eligibility or legal obligations get the official text or a person, never a generated answer.Actions only through tools cost 2The bot can only say something was done when a booking tool confirms it.Take it offline cost 0Switch the AI off.

Cost 0 / 4

Logs are reconstructions: names, times and most wording are invented to match the documented pattern, except lines marked "as reported". Which controls stop which step is a teaching model based on common judgement, not a measurement. Points: 100 when both phases pass first time, 10 less per failed submission.

Built with Llama