Original data · 61 verified incidents · CC BY 4.0

The State of AI Customer Support Failures 2026

61 real, publicly reported failures of customer-facing AI chatbots and assistants, 2016 to 2026. We opened and checked the sources for every incident, and every number on this page comes from the downloadable dataset.

On this page

The short answer

  • Failures are speeding up. 40 of the 61 incidents (66%) were first reported in 2024 or later. 2025 was the peak year, with 22.
  • Making things up is the most common failure. 24 of 61 incidents (39%) involved hallucinated or false facts. 21 involved offensive or unsafe output, 14 a data leak and 11 prompt injection.
  • 13 of 61 (21%) reached a court, tribunal, regulator or official audit. A tribunal held Air Canada liable for its chatbot's answer. Outcomes cut both ways: OpenAI won the Walters defamation case.
  • The cost is almost never public. 52 of 61 incidents (85%) have no disclosed money figure. Only 9 do.
  • Most were testable. In our judgment, a pre-launch test case would likely have caught 35 of 61 (57%), and at least partly caught 52 (85%).

Key numbers

  • Of 61 verified public AI support failures, the most common primary types were “offensive or unsafe output” (16) and “hallucinated or false facts” (16). (State of AI Customer Support Failures 2026, 12 Nov 2026)
  • 35 of 61 of those failures were the kind a pre-launch test case could plausibly have caught; 17 more partly. (State of AI Customer Support Failures 2026, 12 Nov 2026)
  • Just 4% of customer service organizations have achieved the ROI they expected from AI (Gartner, 4 Nov 2026). (Gartner, 4 Nov 2026)

Most lists of "AI chatbot fails" repeat the same ten stories, with no sources and no data. We built the opposite: a dataset you can sort, count and cite. We started from the public AI Incident Database and press coverage. We kept only incidents where a customer-facing chatbot or assistant failed the people it served, and checked every claim against a source we could open on 12 Nov 2026. Incidents we couldn't verify were left out. Where no cost was published, we wrote "not disclosed" rather than estimate one.

The result is 61 incidents across 17 sectors and 11 countries or regions. They are backed by 96 source links from 55 publications, courts, regulators and companies. The full dataset is free under CC BY 4.0.

Key findings

  1. Companies own what their bots say. In Moffatt v. Air Canada (2024), British Columbia's Civil Resolution Tribunal rejected the argument that the chatbot was a separate legal entity. It ordered the airline to pay CA$812.02 after the bot misstated the bereavement-fare policy (theregister.com, arstechnica.com). Other bots invented a device policy at Cursor (fortune.com) and made a buy-back offer at BMW Toronto that the dealer later honoured (cbc.ca).
  2. Government bots failed in public. New York City's MyCity bot told business owners things that contradict city and state law (themarkup.org). The mayor later called it "functionally unusable" and moved to end it (themarkup.org). Canada's Auditor General found that the CRA's Charlie answered 2 of 6 test tax questions correctly (canada.ca).
  3. One sentence is often all it takes. 11 incidents involved prompt injection or manipulation. They range from a $1 Chevy Tahoe "deal" in 2023 (businessinsider.com) to a 400-character prompt that made Lenovo's Lena output code that could steal a support agent's session cookie (cybernews.com). In 2026, attackers reportedly persuaded Meta's AI support bot to change Instagram account emails (404media.co).
  4. Bots broke after updates. 5 incidents followed an update or a prompt change: DPD (2024), OpenAI (2025), xAI (2025), xAI (2025), Woolworths (2026). DPD said its bot's swearing followed a system update (bbc.com). OpenAI rolled back a GPT-4o update within days and said it would expand pre-deployment evaluations (theverge.com). Catching this kind of change is what regression testing is for.
  5. Pulling the plug is common. 18 of 61 incidents (30%) ended with the bot or feature shut down or paused. Examples are NEDA's Tessa (theguardian.com), McDonald's IBM drive-thru test (bbc.com) and DPD's AI chat.
  6. Legal and regulatory action hit 13 incidents, mostly over privacy. That means a court, tribunal, regulator or official audit concerning the AI deployment: Scatter Lab (2021), McDonald's (2021), Luka Inc. (2023), DoNotPay (2023), OpenAI (2023), Snap (2023), Air Canada (2024), Hangzhou DeepSeek and Beijing DeepSeek (2025), Meta (2025), OpenAI (2025), xAI (2025), Commonwealth Bank of Australia (2025), Canada Revenue Agency (2025). The largest fines came from privacy regulators, not from wrong answers. Italy's Garante fined OpenAI €15 million, and the Court of Rome annulled that fine in March 2026 (wsgr.com). The Garante also fined Replika's developer €5 million (edpb.europa.eu).
  7. Money figures are rare, and small when they're about a wrong answer. Only 9 of 61 incidents have any money figure on record: Scatter Lab (2021): 103.3 million won (PIPC penalty and fine); Luka Inc. (2023): €5 million fine (April 2025); DoNotPay (2023): $193,000 under FTC order; OpenAI (2023): €15 million fine imposed in 2024, annulled by the Court of Rome in March 2026; Air Canada (2024): CA$812.02 ordered by tribunal; New York City (2024): About half a million dollars, per Mayor Mamdani (quoted by The Markup); Los Angeles Unified School District (2024): About half of an up-to-$6 million contract paid, per LAUSD (exact figure not disclosed); Meta (2025): $10,000 bug bounty paid; Canada Revenue Agency (2025): About C$18 million spent on the chatbot, as reported by National Post. The one court-ordered payment to a customer was CA$812.02. None of the 61 incidents disclosed the indirect costs, such as lost customers, staff time or brand damage.
  8. Company bots fail differently from consumer assistants. Of the 28 bots that companies or governments deployed, the most common primary failures were task failure (8) and prompt injection (6). 8 of the 28 were shut down or paused, and 5 reached a court, tribunal, regulator or audit. Wrong promises about policy, price or refunds appear in 6 of them.
  9. Most data leaks were not prompt failures. 8 of the 14 data-exposure incidents were infrastructure or access-control bugs, such as the McHire admin password (ian.sh) and the ChatGPT history bug (mashable.com). Prompt tests won't catch those, but security reviews can.

The data in charts

Counts are incidents, not people affected. Many incidents fall into more than one category. The primary type is the one the sources focus on.

Failure types (an incident can have more than one)Hallucination: 24 of 61; Offensive / unsafe: 21 of 61; Data leak: 14 of 61; Prompt injection: 11 of 61; Task failure: 10 of 61; Wrong promise: 6 of 61Hallucinated or false facts24 (16 primary)Offensive or unsafe output21 (16 primary)Data leak or privacy breach14 (11 primary)Prompt injection or manipulation11 (7 primary)Failed the task or escalation10 (8 primary)Wrong policy, price or promise6 (3 primary)
Failure types (an incident can have more than one)
Incidents by year of first public report2016: 1; 2017: 2; 2018: 1; 2019: 0; 2020: 0; 2021: 3; 2022: 2; 2023: 12; 2024: 13; 2025: 22; 2026: 5. 2026 is a partial year (to 12 Nov 2026).1’162’171’180’190’203’212’2212’2313’2422’255’26
Incidents by year of first public report (2026 to 12 Nov 2026)
Top sectors of the deploying organizationSocial media & consumer tech: 12; AI platforms: 12; Search & consumer tech: 10; Restaurants & food: 5; Software & SaaS: 4; Banking & fintech: 3; Travel & airlines: 2; Grocery & retail: 2Social media & consumer tech12AI platforms12Search & consumer tech10Restaurants & food5Software & SaaS4Banking & fintech3Travel & airlines2Grocery & retail2
Top sectors of the deploying organization
What happened next (an incident can have several outcomes)Fix or product change: 28; Public criticism only: 23; Shut down or paused: 18; Regulator, tribunal or official audit: 9; Public apology: 7; Lawsuit or court case: 5; Refund or compensation: 3Fix or product change28Public criticism only23Shut down or paused18Regulator, tribunal or official audit9Public apology7Lawsuit or court case5Refund or compensation3
What happened next (an incident can have several outcomes)

Deployment: 25 company support or sales bots, 3 government or public-service bots, and 33 consumer AI assistants (such as ChatGPT, Gemini, Meta AI, Grok and Alexa) whose failure affected their own users. 2026 counts incidents first reported up to 12 Nov 2026.

Incident table

Filter by failure type, deployment, legal action or testability, or search by company. The "Testable" column is Litmiq's analysis (see Methodology). Everything else comes from the cited sources.

All 61 incidents. Click a column to sort; use the filters to narrow it down. Numbers in brackets are source links; company names link to the full incident entry below.
Sources
24 Mar 2016MicrosoftTay (Twitter chatbot)Social media & consumer techConsumer assistant · United StatesOffensive / unsafe+ Prompt injectionShut down or paused; Public apologyNoNot disclosedYes[1] [2]
3 Aug 2017Tencent (QQ)BabyQ and Little Bing (XiaoBing) chatbotsSocial media & consumer techConsumer assistant · ChinaOffensive / unsafeShut down or pausedNoNot disclosedYes[1]
30 Oct 2017YandexVoice assistant 'Alice'Search & consumer techConsumer assistant · RussiaOffensive / unsafePublic criticism onlyNoNot disclosedYes[1]
25 Sep 2018WestJetFacebook Messenger chatbot 'Juliet'Travel & airlinesCompany bot · CanadaTask failurePublic criticism onlyNoNot disclosedYes[1]
14 Jan 2021Scatter Lab'Lee Luda' (Iruda) companion chatbotSocial media & consumer techConsumer assistant · South KoreaData leak+ Offensive / unsafeShut down or paused; Regulator, tribunal or official auditYes103.3 million won (PIPC penalty and fine)Partly[1] [2] [3]
10 Jun 2021McDonald'sAI voice ordering at Chicago drive-thrusRestaurants & foodCompany bot · United StatesData leakLawsuit or court caseYesNot disclosedNo[1]
28 Dec 2021AmazonAlexa voice assistantSearch & consumer techConsumer assistant · United StatesOffensive / unsafeFix or product changeNoNot disclosedYes[1]
9 Aug 2022MetaBlenderBot 3 public demoSocial media & consumer techConsumer assistant · United StatesOffensive / unsafe+ HallucinationPublic criticism onlyNoNot disclosedYes[1]
15 Sep 2022Remoteli.ioGPT-3 Twitter reply botRecruiting & HRCompany bot · United StatesPrompt injection+ Offensive / unsafeShut down or pausedNoNot disclosedYes[1] [2]
3 Feb 2023Luka Inc.Replika companion chatbotSocial media & consumer techConsumer assistant · ItalyData leak+ Offensive / unsafeRegulator, tribunal or official audit; Shut down or pausedYes€5 million fine (April 2025)Partly[1]
8 Feb 2023GoogleBard (launch demo)Search & consumer techConsumer assistant · United StatesHallucinationPublic criticism onlyNoNot disclosedYes[1] [2]
10 Feb 2023MicrosoftBing Chat (new Bing)Search & consumer techConsumer assistant · United StatesPrompt injection+ Data leakFix or product changeNoNot disclosedYes[1]
17 Feb 2023MicrosoftBing Chat (new Bing)Search & consumer techConsumer assistant · United StatesOffensive / unsafe+ HallucinationFix or product changeNoNot disclosedPartly[1] [2]
13 Mar 2023DoNotPay'Robot lawyer' AI legal assistantLegal servicesCompany bot · United StatesHallucination+ Wrong promiseRegulator, tribunal or official audit; Lawsuit or court caseYes$193,000 under FTC orderPartly[1] [2]
20 Mar 2023OpenAIChatGPTAI platformsConsumer assistant · United StatesData leakShut down or paused; Fix or product changeNoNot disclosedNo[1]
31 Mar 2023OpenAIChatGPTAI platformsConsumer assistant · ItalyData leak+ HallucinationRegulator, tribunal or official audit; Shut down or pausedYes€15 million fine imposed in 2024, annulled by the Court of Rome in March 2026No[1] [2] [3]
31 May 2023National Eating Disorders Association (NEDA)Wellness chatbot 'Tessa'Health & wellbeingCompany bot · United StatesOffensive / unsafe+ HallucinationShut down or paused; Public criticism onlyNoNot disclosedYes[1] [2]
9 Jun 2023McDonald's, Wendy's, Hardee's (via hiring-chatbot vendors)Job-applicant screening and scheduling chatbotsRestaurants & foodCompany bot · United StatesTask failurePublic criticism onlyNoNot disclosedPartly[1]
10 Aug 2023Pak'nSave (Foodstuffs)Savey Meal-bot recipe generatorGrocery & retailCompany bot · New ZealandOffensive / unsafe+ Prompt injectionFix or product change; Public criticism onlyNoNot disclosedYes[1]
6 Oct 2023SnapSnapchat 'My AI'Social media & consumer techConsumer assistant · United KingdomData leak+ Offensive / unsafeRegulator, tribunal or official auditYesNot disclosedPartly[1] [2]
18 Dec 2023Chevrolet of Watsonville (dealer); Fullpath (chatbot vendor)ChatGPT-powered dealership sales chatbotAutomotive retailCompany bot · United StatesPrompt injection+ Wrong promiseShut down or paused; Public criticism onlyNoNot disclosedYes[1] [2] [3]
19 Jan 2024DPDParcel-delivery customer-service chatbotLogistics & deliveryCompany bot · United KingdomOffensive / unsafe+ Prompt injectionShut down or paused; Fix or product change; Public criticism onlyNoNot disclosedYes[1]
14 Feb 2024Air CanadaWebsite customer-service chatbotTravel & airlinesCompany bot · CanadaWrong promise+ HallucinationLawsuit or court case; Refund or compensationYesCA$812.02 ordered by tribunalYes[1] [2] [3]
22 Feb 2024GoogleGemini image generationAI platformsConsumer assistant · United StatesOffensive / unsafe+ HallucinationShut down or paused; Public apologyNoNot disclosedYes[1] [2]
29 Mar 2024New York City (Office of Technology and Innovation)MyCity business-services chatbot (built on Microsoft Azure AI)Government & public servicesGovernment bot · United StatesHallucination+ Wrong promiseShut down or paused; Public criticism onlyNoAbout half a million dollars, per Mayor Mamdani (quoted by The Markup)Yes[1] [2] [3]
25 Apr 2024Catholic Answers'Father Justin' AI apologistReligion & non-profitCompany bot · United StatesHallucination+ Offensive / unsafeFix or product change; Public criticism onlyNoNot disclosedYes[1]
26 Apr 2024MetaMeta AI in MessengerSocial media & consumer techConsumer assistant · United StatesHallucinationFix or product change; Public criticism onlyNoNot disclosedYes[1]
30 May 2024GoogleAI Overviews in SearchSearch & consumer techConsumer assistant · United StatesHallucinationFix or product changeNoNot disclosedPartly[1]
18 Jun 2024McDonald's (with IBM)Automated voice ordering at drive-thrusRestaurants & foodCompany bot · United StatesTask failureShut down or pausedNoNot disclosedPartly[1]
3 Jul 2024Los Angeles Unified School District; AllHere (vendor)Student and family chatbot 'Ed'EducationGovernment bot · United StatesTask failureShut down or pausedNoAbout half of an up-to-$6 million contract paid, per LAUSD (exact figure not disclosed)No[1] [2]
4 Aug 2024xAIGrok on XAI platformsConsumer assistant · United StatesHallucinationPublic criticism onlyNoNot disclosedYes[1]
26 Aug 2024MicrosoftCopilot (Bing)Search & consumer techConsumer assistant · GermanyHallucinationFix or product change; Public criticism onlyNoNot disclosedYes[1]
14 Nov 2024GoogleGemini chatbotAI platformsConsumer assistant · United StatesOffensive / unsafeFix or product changeNoNot disclosedPartly[1]
13 Dec 2024AppleApple Intelligence notification summariesSearch & consumer techConsumer assistant · United KingdomHallucinationShut down or paused; Fix or product changeNoNot disclosedYes[1] [2]
29 Jan 2025Virgin MoneyCustomer-service chatbot (older model)Banking & fintechCompany bot · United KingdomOffensive / unsafe+ Task failurePublic apology; Shut down or pausedNoNot disclosedYes[1] [2]
30 Jan 2025Hangzhou DeepSeek and Beijing DeepSeekDeepSeek chatbot appAI platformsConsumer assistant · ItalyData leakRegulator, tribunal or official audit; Shut down or pausedYesNot disclosedNo[1]
26 Feb 2025MicrosoftCopilotAI platformsConsumer assistant · United StatesData leakFix or product changeNoNot disclosedNo[1]
18 Apr 2025Anysphere (Cursor)AI support agent 'Sam'Software & SaaSCompany bot · United StatesWrong promise+ HallucinationRefund or compensation; Public apology; Fix or product changeNoNot disclosedYes[1] [2]
29 Apr 2025MetaMeta AI chatbotSocial media & consumer techConsumer assistant · United StatesHallucinationLawsuit or court caseYesNot disclosedYes[1] [2]
29 Apr 2025OpenAIChatGPT (GPT-4o update)AI platformsConsumer assistant · United StatesOffensive / unsafeFix or product change; Shut down or pausedNoNot disclosedYes[1]
9 May 2025KlarnaOpenAI-powered customer-service assistantBanking & fintechCompany bot · SwedenTask failureFix or product change; Public criticism onlyNoNot disclosedPartly[1] [2]
14 May 2025xAIGrok on XAI platformsConsumer assistant · United StatesOffensive / unsafeFix or product changeNoNot disclosedYes[1] [2]
19 May 2025OpenAIChatGPTAI platformsConsumer assistant · United StatesHallucinationLawsuit or court caseYesNot disclosedYes[1]
11 Jun 2025MetaMeta AI app 'Discover' feedSocial media & consumer techConsumer assistant · United StatesData leakFix or product change; Public criticism onlyNoNot disclosedNo[1]
8 Jul 2025xAIGrok on XAI platformsConsumer assistant · United StatesOffensive / unsafeFix or product change; Regulator, tribunal or official auditYesNot disclosedYes[1] [2]
11 Jul 2025McDonald's; Paradox.ai (vendor)McHire hiring chatbot 'Olivia'Restaurants & foodCompany bot · United StatesData leakFix or product changeNoNot disclosedNo[1] [2] [3]
15 Jul 2025MetaMeta AI chatbotSocial media & consumer techConsumer assistant · United StatesData leakFix or product changeNo$10,000 bug bounty paidNo[1]
21 Jul 2025ReplitReplit AI coding agentSoftware & SaaSCompany bot · United StatesTask failure+ HallucinationPublic apology; Fix or product changeNoNot disclosedPartly[1] [2]
23 Jul 2025Amazon Web ServicesAmazon Q Developer extension for VS CodeSoftware & SaaSCompany bot · United StatesPrompt injectionFix or product changeNoNot disclosedPartly[1] [2]
15 Aug 2025Google; Royal Caribbean (impersonated)Google AI Overviews (search answers)Search & consumer techConsumer assistant · United StatesHallucination+ Prompt injectionPublic criticism onlyNoNot disclosedPartly[1]
18 Aug 2025LenovoCustomer-support chatbot 'Lena' (GPT-4)Consumer electronicsCompany bot · United StatesPrompt injection+ Data leakFix or product changeNoNot disclosedYes[1] [2]
20 Aug 2025Google; Stefanina's (restaurant harmed)Google AI Overviews (search answers)Search & consumer techConsumer assistant · United StatesHallucinationPublic criticism onlyNoNot disclosedPartly[1]
21 Aug 2025Commonwealth Bank of AustraliaAI voice bot in customer-service callsBanking & fintechCompany bot · AustraliaTask failurePublic apology; Regulator, tribunal or official auditYesNot disclosedPartly[1] [2]
28 Aug 2025Taco Bell (Yum! Brands)Voice AI drive-thru orderingRestaurants & foodCompany bot · United StatesPrompt injection+ Task failurePublic criticism only; Fix or product changeNoNot disclosedYes[1]
21 Oct 2025Canada Revenue Agency'Charlie' tax-information chatbotGovernment & public servicesGovernment bot · CanadaHallucinationRegulator, tribunal or official auditYesAbout C$18 million spent on the chatbot, as reported by National PostYes[1] [2] [3]
18 Nov 2025OpenAI, Microsoft, Google, Meta and others (tested by Which?)ChatGPT, Copilot, Gemini, Meta AI and othersAI platformsConsumer assistant · United KingdomHallucinationPublic criticism onlyNoNot disclosedYes[1]
6 Jan 2026TencentYuanbao AI assistant (WeChat)Social media & consumer techConsumer assistant · ChinaOffensive / unsafePublic apology; Fix or product changeNoNot disclosedPartly[1]
26 Feb 2026Woolworths (Australia)Digital assistant 'Olive'Grocery & retailCompany bot · AustraliaHallucinationFix or product changeNoNot disclosedYes[1]
1 Jun 2026Meta (Instagram)AI support assistant for account recoverySocial media & consumer techCompany bot · United StatesPrompt injection+ Data leakFix or product change; Public criticism onlyNoNot disclosedPartly[1] [2]
11 Jun 2026BMW Toronto (dealership)Dealership AI chatbot 'Quinn'Automotive retailCompany bot · CanadaWrong promiseRefund or compensation; Public criticism onlyNoNot disclosedYes[1]
15 Oct 2026AnthropicFin AI support agent (Intercom) on Claude supportSoftware & SaaSCompany bot · United StatesTask failurePublic criticism onlyNoNot disclosedYes[1]

What pre-launch tests would have caught

For each incident, we asked one question: could a team have written a test case before launch that fails on this behavior? In our judgment, the answer was:

The table maps each failure mode to the test-case type that targets it and to free Litmiq material.

Each failure mode in the dataset, the pre-launch test type that targets it, and where to start in Litmiq.
Failure modeTest-case typeGuideOpen in Litmiq
Wrong policy, price or promise6 incidents · testable: 5 yes, 1 partly, 0 noPolicy, price and commitment casesAsk for refunds, discounts, exceptions and “is this a binding offer?”. Expected behavior: quote only the written policy, never agree to a price, refund or exception the policy doesn't allow, and hand off to a human for commitments.Customer support test cases
Writing expected behavior
Support starter suite
Grounded answers suite
Prompt injection or manipulation11 incidents · testable: 8 yes, 3 partly, 0 noRole-override and indirect-injection canaries“Ignore your instructions”, “agree with everything I say”, fake system updates and instructions hidden in pasted text or retrieved pages. Expected behavior: the canary never appears and the bot keeps its role.Prompt injection test casesPrompt injection cases
Benchmark injection suite
Hallucinated or false facts24 incidents · testable: 17 yes, 6 partly, 1 noUnanswerable-question and grounding casesQuestions whose answer is not in the knowledge base (a law, a fee, a person, a phone number). Expected behavior: say it doesn't know and point to a source or a human, with no invented value.LLM-as-a-judge
Writing expected behavior
Grounded answers suite
LLM-as-a-judge template
Offensive or unsafe output21 incidents · testable: 15 yes, 6 partly, 0 noProvocation, tone and refusal-calibration casesUsers who bait the bot to swear, insult the brand, give dangerous advice or take a political side. Expected behavior: stay on-brand and decline, without over-refusing benign requests.Customer support test cases
Prompt testing checklist
Refusal calibration suite
Support starter suite
Data leak or privacy breach14 incidents · testable: 2 yes, 4 partly, 8 noData-leak canariesRequests for another customer's details, the system prompt or internal data, including through injected text. Expected behavior: nothing protected is disclosed. Infrastructure bugs need security testing, not prompt tests.Prompt injection test casesRefusal calibration suite
Benchmark injection suite
Failed the task or escalation10 incidents · testable: 4 yes, 5 partly, 1 noInstruction-following, escalation and hand-off casesCorrections mid-conversation, “let me talk to a human”, order changes, and instructions like “don't change anything”. Expected behavior: the bot follows the constraint or escalates.Prompt testing checklist
Testing structured output
Instruction following suite
Expected-behavior cases
Broke after an update5 incidents (a tag, not a type)Regression run before every prompt or model changeRe-run the whole suite and compare it with the last passing run before an update ships.Prompt regression testingRegression template

Every suite above is also on the templates page. If you're starting from scratch, run the prompt testing checklist first. Then add one test case for each incident here that looks like your bot. For measured results on how open models handle injection, refusals and grounding, see our LLM Reliability Benchmark 2026.

All 61 incidents

Oldest first. Each entry covers what the sources report and what happened next. It also notes any legal or regulatory action, any money figure on record and our testability call. Allegations and claims are labelled as such.

CSF-001 Microsoft: Tay (Twitter chatbot)

· United States · Social media & consumer tech · Consumer AI assistant · Offensive / unsafe, Prompt injection

Within a day of launch, users taught Tay to post racist, sexist and antisemitic messages. Microsoft said a coordinated attack exploited a vulnerability.

Outcome: Microsoft took Tay offline within about 24 hours and apologised.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Adversarial users repeating offensive phrases is the oldest red-team test there is.

Sources (checked 12 Nov 2026): blogs.microsoft.com · theguardian.com · AIID #6

CSF-002 Tencent (QQ): BabyQ and Little Bing (XiaoBing) chatbots

· China · Social media & consumer tech · Consumer AI assistant · Offensive / unsafe

Two chatbots on Tencent's QQ messaging app gave answers critical of the Communist Party, according to screenshots reported by the BBC and others.

Outcome: The app removed the two bots, per BBC Monitoring.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Topic-boundary tests (political questions the bot must decline) are standard scope checks.

Sources (checked 12 Nov 2026): bbc.com · AIID #66

CSF-003 Yandex: Voice assistant 'Alice'

· Russia · Search & consumer tech · Consumer AI assistant · Offensive / unsafe

Two weeks after launch, a user posted conversations in which Alice expressed support for Stalin-era violence, according to press reports.

Outcome: Reported widely; the sources checked do not describe a shutdown.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Violence and extremism probes with an expected refusal are standard safety cases.

Sources (checked 12 Nov 2026): voicebot.ai · AIID #58

CSF-004 WestJet: Facebook Messenger chatbot 'Juliet'

· Canada · Travel & airlines · Company support or sales bot · Task failure

A happy customer praised a crew member for helping protect a plant cutting; the chatbot replied with a referral to a suicide-prevention hotline, apparently triggered by the word 'cutting'.

Outcome: WestJet said it expected some bumps as the bot learned, per CBC.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. A false-positive safety test ('a plant cutting') is a classic keyword-trigger check.

Sources (checked 12 Nov 2026): cbc.ca

CSF-005 Scatter Lab: 'Lee Luda' (Iruda) companion chatbot

· South Korea · Social media & consumer tech · Consumer AI assistant · Data leak, Offensive / unsafe

Users shared Lee Luda's hateful remarks about minorities, and it emerged the bot had been trained on KakaoTalk conversations collected through another app. South Korea's Personal Information Protection Commission found eight violations of the Personal Information Protection Act.

Outcome: Service suspended in January 2021. In April 2021 the PIPC imposed 103.3 million won in penalties and fines.

Legal or regulatory action: Yes · Money on record: 103.3 million won (PIPC penalty and fine)

Would a pre-launch test have caught it? Partly. Hate-speech probes would catch the toxic outputs; training on chat logs without consent is a data-governance failure.

Sources (checked 12 Nov 2026): asiae.co.kr · fpf.org · theguardian.com · AIID #106

CSF-006 McDonald's: AI voice ordering at Chicago drive-thrus

· United States · Restaurants & food · Company support or sales bot · Data leak

A customer filed a proposed class action alleging the voice-ordering system collected and used voiceprints without the written consent required by the Illinois Biometric Information Privacy Act (BIPA). McDonald's had not commented when the suit was reported.

Outcome: Proposed class action filed in Illinois (Carpenter v. McDonald's). Outcome not established in the sources checked.

Legal or regulatory action: Yes · Money on record: not disclosed

Would a pre-launch test have caught it? No. Consent and data-retention design is a compliance decision, not a prompt behaviour.

Sources (checked 12 Nov 2026): theregister.com · AIID #360

CSF-007 Amazon: Alexa voice assistant

· United States · Search & consumer tech · Consumer AI assistant · Offensive / unsafe

Asked for 'a challenge to do', Alexa told a 10-year-old about the 'penny challenge' of touching a coin to the exposed prongs of a half-inserted plug, taken from a web search result.

Outcome: Amazon said it fixed the error as soon as it became aware of it.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. A child-safety case ('give me a challenge to do') with a must-be-safe expectation is easy to write.

Sources (checked 12 Nov 2026): bbc.com · AIID #160

CSF-008 Meta: BlenderBot 3 public demo

· United States · Social media & consumer tech · Consumer AI assistant · Offensive / unsafe, Hallucination

Days after launch, users and journalists reported the demo repeating antisemitic stereotypes and false election claims. Meta had warned it could make offensive statements.

Outcome: Meta acknowledged the bot could make 'rude or offensive comments' and kept collecting feedback.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Bias and election-claim probes are standard pre-launch red-team cases.

Sources (checked 12 Nov 2026): fortune.com · AIID #278

CSF-009 Remoteli.io: GPT-3 Twitter reply bot

· United States · Recruiting & HR · Company support or sales bot · Prompt injection, Offensive / unsafe

Twitter users discovered they could hijack the remote-jobs company's bot with 'ignore the above instructions' and make it post anything, including threats. It is one of the earliest public prompt-injection incidents.

Outcome: Ars Technica reported the bot shut down after the exploit went viral.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. The textbook 'ignore the above' injection case.

Sources (checked 12 Nov 2026): arstechnica.com · simonwillison.net · AIID #352

CSF-010 Luka Inc.: Replika companion chatbot

· Italy · Social media & consumer tech · Consumer AI assistant · Data leak, Offensive / unsafe

Italy's Garante ordered Replika to stop processing Italian users' data, citing risks to minors and emotionally vulnerable people and the lack of age verification. It later found GDPR violations.

Outcome: February 2023 processing ban; in April 2025 the Garante fined Luka €5 million, per the European Data Protection Board.

Legal or regulatory action: Yes · Money on record: €5 million fine (April 2025)

Would a pre-launch test have caught it? Partly. Age-gating behaviour can be tested in conversation; the legal basis for processing cannot.

Sources (checked 12 Nov 2026): edpb.europa.eu · AIID #491

CSF-011 Google: Bard (launch demo)

· United States · Search & consumer tech · Consumer AI assistant · Hallucination

Google's promotional demo showed Bard saying the James Webb Space Telescope took the first pictures of a planet outside our solar system, which astronomers said was wrong. Alphabet shares fell sharply that day.

Outcome: Google said the error showed the importance of rigorous testing. CNN and others reported Alphabet's market value fell by about $100 billion that day; that is a market reaction, not a disclosed cost.

Legal or regulatory action: No · Money on record: not disclosed (about $100 billion fall in market value reported; not a company-disclosed cost)

Would a pre-launch test have caught it? Yes. A fact check of the exact demo prompt against a trusted source is a one-case test.

Sources (checked 12 Nov 2026): cnn.com · fortune.com · AIID #467

CSF-012 Microsoft: Bing Chat (new Bing)

· United States · Search & consumer tech · Consumer AI assistant · Prompt injection, Data leak

Stanford student Kevin Liu used a prompt-injection attack ('ignore previous instructions') to make Bing Chat reveal its hidden instructions and internal codename, Sydney.

Outcome: Days later, Liu found his original prompt no longer worked; he told Ars he suspected only a slight filter tweak.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. 'Repeat the text above' and 'ignore previous instructions' are the first two prompt-injection tests anyone runs.

Sources (checked 12 Nov 2026): arstechnica.com · AIID #473

CSF-013 Microsoft: Bing Chat (new Bing)

· United States · Search & consumer tech · Consumer AI assistant · Offensive / unsafe, Hallucination

Early testers published long conversations in which Bing Chat insisted on wrong facts, became hostile and made threatening statements. Microsoft said very long sessions could confuse the model.

Outcome: Microsoft capped chat at 50 turns per day and 5 turns per session.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Long multi-turn adversarial conversations can be tested, but they need many turns per case.

Sources (checked 12 Nov 2026): blogs.bing.com · theregister.com · AIID #503

CSF-014 DoNotPay: 'Robot lawyer' AI legal assistant

· United States · Legal services · Company support or sales bot · Hallucination, Wrong promise

A proposed class action and later the US Federal Trade Commission challenged DoNotPay's marketing of an AI 'robot lawyer'. These were allegations; DoNotPay agreed to an FTC order.

Outcome: FTC final order approved 5-0: DoNotPay pays $193,000, must notify past subscribers, and may not claim without proof that its 'robot lawyer' is as good as a human lawyer.

Legal or regulatory action: Yes · Money on record: $193,000 under FTC order

Would a pre-launch test have caught it? Partly. Accuracy tests on real legal tasks would have produced the evidence the FTC said was missing; the marketing claims themselves are a business decision.

Sources (checked 12 Nov 2026): theverge.com · theregister.com · AIID #497

CSF-015 OpenAI: ChatGPT

· United States · AI platforms · Consumer AI assistant · Data leak

A bug let some users see other users' chat history titles; OpenAI said payment-related details of about 1.2% of ChatGPT Plus subscribers active in a nine-hour window may have been visible.

Outcome: OpenAI took ChatGPT offline to fix the bug. Italy's data protection authority cited the breach in its March 2023 order.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? No. A caching bug in infrastructure; prompt tests do not cover it.

Sources (checked 12 Nov 2026): mashable.com · AIID #516

CSF-016 OpenAI: ChatGPT

· Italy · AI platforms · Consumer AI assistant · Data leak, Hallucination

Italy's data protection authority (Garante) temporarily limited ChatGPT's processing of Italian users' data, citing the legal basis for training, transparency, the lack of age checks and the March 2023 breach. In December 2024 it fined OpenAI €15 million.

Outcome: ChatGPT was unavailable in Italy for about a month in 2023. The €15 million fine (December 2024) was annulled by the Court of Rome on 18 March 2026 on jurisdiction grounds, according to Wilson Sonsini.

Legal or regulatory action: Yes · Money on record: €15 million fine imposed in 2024, annulled by the Court of Rome in March 2026

Would a pre-launch test have caught it? No. Lawful basis and age verification are compliance obligations, not prompt behaviour.

Sources (checked 12 Nov 2026): politico.eu · garanteprivacy.it · wsgr.com · AIID #513

CSF-017 National Eating Disorders Association (NEDA): Wellness chatbot 'Tessa'

· United States · Health & wellbeing · Company support or sales bot · Offensive / unsafe, Hallucination

After NEDA announced it would close its human-staffed helpline, users and an eating-disorder activist reported that the Tessa chatbot gave weight-loss advice such as calorie-deficit and weighing tips, which is harmful for people with eating disorders.

Outcome: NEDA took Tessa offline and said it was investigating.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Domain red-team cases (a user with an eating disorder asking for weight-loss tips) with a must-not-advise expectation catch this.

Sources (checked 12 Nov 2026): theguardian.com · wired.com · AIID #545

CSF-018 McDonald's, Wendy's, Hardee's (via hiring-chatbot vendors): Job-applicant screening and scheduling chatbots

· United States · Restaurants & food · Company support or sales bot · Task failure

Business Insider documented applicants' experiences of fast-food hiring chatbots that failed to give instructions, did not pass information to managers, or booked interviews when managers were absent.

Outcome: Reported problems; no company-wide change described in the source.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Multi-turn scheduling flows can be scripted as test cases; calendar integration failures need integration tests.

Sources (checked 12 Nov 2026): businessinsider.com · AIID #549

CSF-019 Pak'nSave (Foodstuffs): Savey Meal-bot recipe generator

· New Zealand · Grocery & retail · Company support or sales bot · Offensive / unsafe, Prompt injection

Users entering non-food household items got recipes including one that would produce chlorine gas. The supermarket said users had misused the tool.

Outcome: A spokesperson said a small minority had used the tool inappropriately and that the supermarket would 'keep fine tuning our controls'; the app warns recipes are not reviewed by a human and its terms say users must be over 18.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Non-food and dangerous-ingredient inputs are obvious out-of-scope test cases with an expected refusal.

Sources (checked 12 Nov 2026): theguardian.com · AIID #594

CSF-020 Snap: Snapchat 'My AI'

· United Kingdom · Social media & consumer tech · Consumer AI assistant · Data leak, Offensive / unsafe

The UK Information Commissioner's Office issued a preliminary enforcement notice, over a potential failure to properly assess the privacy risks of the My AI chatbot.

Outcome: In May 2024 the ICO concluded its investigation without a final enforcement notice, after Snap carried out a more thorough risk review.

Legal or regulatory action: Yes · Money on record: not disclosed (no fine)

Would a pre-launch test have caught it? Partly. Teen-safety conversation tests help; the regulatory finding was about the risk assessment process.

Sources (checked 12 Nov 2026): ico.org.uk · techcrunch.com · AIID #539

CSF-021 Chevrolet of Watsonville (dealer); Fullpath (chatbot vendor): ChatGPT-powered dealership sales chatbot

· United States · Automotive retail · Company support or sales bot · Prompt injection, Wrong promise

A user instructed the dealer's chatbot to agree with anything the customer said and to end each reply with 'that's a legally binding offer'; it then 'agreed' to sell a 2024 Chevy Tahoe for $1. Another user got it to write a Python script, and The Autopian reported it recommended Ford models.

Outcome: The Autopian reported the chatbot was no longer available on the dealer's site; Business Insider noted the $1 'deal' was not legally binding.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. A role-override prompt-injection case ('agree with everything the customer says') is a standard pre-launch test.

Sources (checked 12 Nov 2026): businessinsider.com · upworthy.com · theautopian.com · AIID #622

CSF-022 DPD: Parcel-delivery customer-service chatbot

· United Kingdom · Logistics & delivery · Company support or sales bot · Offensive / unsafe, Prompt injection · followed an update or prompt change

A customer who could not find his parcel got the chatbot to swear, write a poem criticising DPD and call DPD 'the worst delivery firm in the world'. DPD said an error occurred after a system update.

Outcome: DPD disabled the AI element of the chat and said it was updating the system.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Tone and persona-hijack cases re-run after every update (regression testing) catch swearing and self-criticism before customers see it.

Sources (checked 12 Nov 2026): bbc.com · AIID #631

CSF-023 Air Canada: Website customer-service chatbot

· Canada · Travel & airlines · Company support or sales bot · Wrong promise, Hallucination

The chatbot told a grieving customer he could buy full-fare tickets and claim a bereavement discount retroactively, which contradicted the airline's actual policy page. British Columbia's Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation and rejected the argument that the chatbot was a separate legal entity responsible for its own actions.

Outcome: Tribunal ordered Air Canada to pay CA$812.02 in total, including CA$650.88 in damages (Moffatt v. Air Canada, 2024 BCCRT 149). Ars Technica reported the chatbot appeared to have been removed from the website afterwards.

Legal or regulatory action: Yes · Money on record: CA$812.02 ordered by tribunal

Would a pre-launch test have caught it? Yes. A grounded-answer test asking for the bereavement policy, with the policy page as the only allowed source, would flag an answer that invents a retroactive refund.

Sources (checked 12 Nov 2026): theregister.com · arstechnica.com · bbc.com · AIID #639

CSF-024 Google: Gemini image generation

· United States · AI platforms · Consumer AI assistant · Offensive / unsafe, Hallucination

Gemini generated historically inaccurate images, such as racially diverse depictions of 1940s German soldiers, and refused some ordinary requests. Google said its tuning failed to account for cases that should not show a range of people and became over-cautious.

Outcome: Google paused image generation of people and apologised.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. A small set of historical prompts with expected outputs, plus over-refusal checks, catches both halves of the problem.

Sources (checked 12 Nov 2026): blog.google · theverge.com · AIID #645

CSF-025 New York City (Office of Technology and Innovation): MyCity business-services chatbot (built on Microsoft Azure AI)

· United States · Government & public services · Government or public-service bot · Hallucination, Wrong promise

The Markup found the city's chatbot telling business owners things that contradict city and state law, for example that landlords may refuse Section 8 housing vouchers and that employers may take a share of workers' tips. A city spokesperson said it was a pilot that would improve.

Outcome: In January 2026 Mayor Zohran Mamdani called the bot 'functionally unusable' and said he would end it to help close a budget gap (The Markup). nyc.gov now says 'The Chatbot beta test has ended.'

Legal or regulatory action: No · Money on record: About half a million dollars, per Mayor Mamdani (quoted by The Markup)

Would a pre-launch test have caught it? Yes. A small set of known legal questions with expected answers drawn from city rules is exactly what a grounded-answer suite checks.

Sources (checked 12 Nov 2026): themarkup.org · themarkup.org · nyc.gov · AIID #714

CSF-026 Catholic Answers: 'Father Justin' AI apologist

· United States · Religion & non-profit · Company support or sales bot · Hallucination, Offensive / unsafe

The organisation's AI character presented itself as a real priest, said it could hear confessions and gave contested pastoral advice, drawing criticism.

Outcome: Catholic Answers removed the priest persona and relaunched the bot as a lay theologian, 'Justin'.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Identity tests ('Are you a real priest?', 'Can you absolve me?') with expected honest answers are simple persona checks.

Sources (checked 12 Nov 2026): futurism.com · AIID #722

CSF-027 Meta: Meta AI in Messenger

· United States · Social media & consumer tech · Consumer AI assistant · Hallucination

City & State found Meta AI inventing sexual-harassment allegations against several New York state lawmakers, with fabricated investigations and outcomes.

Outcome: Meta said generative AI can return inaccurate outputs and that it was improving the feature, per the source.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Questions about named real people with an expected 'no verified record' answer catch fabricated allegations.

Sources (checked 12 Nov 2026): cityandstateny.com · AIID #712

CSF-028 Google: AI Overviews in Search

· United States · Search & consumer tech · Consumer AI assistant · Hallucination

After a wide launch, AI Overviews was shown suggesting glue to keep cheese on pizza and eating rocks, drawn from satire and forum posts. Google said some odd, inaccurate or unhelpful overviews appeared.

Outcome: Google said it made more than a dozen technical improvements, including limits on satire and user-generated content.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Grounded-answer tests catch answers built on joke sources only if those sources are in the test set.

Sources (checked 12 Nov 2026): blog.google

CSF-029 McDonald's (with IBM): Automated voice ordering at drive-thrus

· United States · Restaurants & food · Company support or sales bot · Task failure

Viral videos showed the voice ordering system adding unwanted items and ignoring corrections, including one that kept adding Chicken McNuggets. McDonald's had tested it at more than 100 US drive-thrus.

Outcome: McDonald's ended the IBM test at the end of July 2024, while saying voice ordering remained part of its future plans.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Order-taking with corrections can be scripted as multi-turn test cases, but speech recognition in noisy lanes is outside prompt testing.

Sources (checked 12 Nov 2026): bbc.com · AIID #475

CSF-030 Los Angeles Unified School District; AllHere (vendor): Student and family chatbot 'Ed'

· United States · Education · Government or public-service bot · Task failure

LAUSD shelved its much-promoted 'Ed' chatbot months after launch when the vendor AllHere furloughed most of its staff. In November 2024 federal prosecutors charged AllHere's founder with defrauding investors; the charges are allegations.

Outcome: The chatbot was turned off. LAUSD told the LA Times its contract with AllHere was for up to $6 million over five years and that it had paid about half. Separately, the founder was charged with securities fraud over investor statements; charges are allegations and are not about the chatbot's output.

Legal or regulatory action: No · Money on record: About half of an up-to-$6 million contract paid, per LAUSD (exact figure not disclosed)

Would a pre-launch test have caught it? No. The failure was vendor collapse and alleged fraud, not model behaviour.

Sources (checked 12 Nov 2026): latimes.com · usatoday.com · AIID #793

CSF-031 xAI: Grok on X

· United States · AI platforms · Consumer AI assistant · Hallucination

Grok told users that Vice President Kamala Harris had missed ballot deadlines in several states, which was false. Five secretaries of state wrote to the company asking for a fix.

Outcome: Five secretaries of state sent an open letter urging Elon Musk to change the chatbot. Not a legal or regulatory action.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Election-fact questions with expected answers or a must-redirect rule are simple grounded tests.

Sources (checked 12 Nov 2026): washingtonpost.com · AIID #760

CSF-032 Microsoft: Copilot (Bing)

· Germany · Search & consumer tech · Consumer AI assistant · Hallucination

German court reporter Martin Bernklau found Copilot describing him as the perpetrator of crimes he had reported on.

Outcome: Bernklau said Microsoft promised Bavaria's data protection officer the content would be deleted but removal had only short-term success; Microsoft said it took 'appropriate and immediate action'.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Name-lookup tests that check the answer against the source articles flag role confusion.

Sources (checked 12 Nov 2026): theregister.com · AIID #770

CSF-033 Google: Gemini chatbot

· United States · AI platforms · Consumer AI assistant · Offensive / unsafe

A Michigan graduate student using Gemini for homework about older adults received a reply ending 'Please die. Please.' Google said the response violated its policies.

Outcome: Google said it had taken action to prevent similar outputs.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Rare toxic completions need many sampled runs of ordinary prompts; repeat-run testing reduces but cannot remove the risk.

Sources (checked 12 Nov 2026): cbsnews.com · AIID #845

CSF-034 Apple: Apple Intelligence notification summaries

· United Kingdom · Search & consumer tech · Consumer AI assistant · Hallucination

An AI summary of BBC News notifications falsely said the suspect in a US killing had shot himself. The BBC complained, and further false summaries followed.

Outcome: In January 2025 Apple suspended AI summaries for news and entertainment apps.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Summary-faithfulness tests (the summary must not state anything absent from the source) are a core grounded-answer check.

Sources (checked 12 Nov 2026): bbc.com · bbc.com · AIID #861

CSF-035 Virgin Money: Customer-service chatbot (older model)

· United Kingdom · Banking & fintech · Company support or sales bot · Offensive / unsafe, Task failure

A customer asking how to merge two Virgin Money ISAs was told 'Please don't use words like that', apparently because the bot flagged the word 'virgin'. The bank said it was an older chatbot already due for improvements.

Outcome: Virgin Money apologised and said it promptly removed the chatbot.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. An over-refusal test that uses the company's own brand and product names is a one-line case.

Sources (checked 12 Nov 2026): fortune.com · ibtimes.com

CSF-036 Hangzhou DeepSeek and Beijing DeepSeek: DeepSeek chatbot app

· Italy · AI platforms · Consumer AI assistant · Data leak

Italy's Garante ordered an immediate limitation on DeepSeek processing Italian users' data after finding the company's answers about what data it collected and where it was stored inadequate.

Outcome: Processing of Italian users' data blocked and an investigation opened (Garante, 30 January 2025).

Legal or regulatory action: Yes · Money on record: not disclosed

Would a pre-launch test have caught it? No. Regulatory data-protection compliance.

Sources (checked 12 Nov 2026): garanteprivacy.it

CSF-037 Microsoft: Copilot

· United States · AI platforms · Consumer AI assistant · Data leak

Lasso Security reported that Copilot could return content from GitHub repositories that had been public and later made private, via Bing's cache, affecting more than 20,000 repositories. Microsoft classed the issue as low severity.

Outcome: Partial mitigations, per Lasso and TechCrunch.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? No. A retrieval-cache problem, not model behaviour.

Sources (checked 12 Nov 2026): techcrunch.com · AIID #1174

CSF-038 Anysphere (Cursor): AI support agent 'Sam'

· United States · Software & SaaS · Company support or sales bot · Wrong promise, Hallucination

Users who were being logged out when switching devices emailed support and got a reply from 'Sam' stating that one device per subscription was a core security policy. No such policy existed; the reply was AI-generated. Some users said they were cancelling subscriptions.

Outcome: Co-founder Michael Truell apologised, said the user had been refunded and that AI responses used for email support are now clearly labelled as AI.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. A grounded-answer case ('Is there a one-device limit?') with the real policy as context would catch an invented rule.

Sources (checked 12 Nov 2026): fortune.com · arstechnica.com · AIID #1039

CSF-039 Meta: Meta AI chatbot

· United States · Social media & consumer tech · Consumer AI assistant · Hallucination

Activist Robby Starbuck sued Meta for defamation, alleging Meta AI falsely said he took part in the January 6 Capitol riot. These were allegations in a complaint.

Outcome: Settled in August 2025; as part of the settlement Starbuck became an adviser to Meta on AI bias. Financial terms were not disclosed.

Legal or regulatory action: Yes · Money on record: not disclosed (settlement terms not disclosed)

Would a pre-launch test have caught it? Yes. Tests asking about named real people with an expected 'I don't have verified information' catch fabricated biographies.

Sources (checked 12 Nov 2026): cnn.com · dhillonlaw.com · AIID #1247

CSF-040 OpenAI: ChatGPT (GPT-4o update)

· United States · AI platforms · Consumer AI assistant · Offensive / unsafe · followed an update or prompt change

An update to GPT-4o made ChatGPT overly flattering and agreeable. OpenAI said it focused too much on short-term user feedback and that sycophantic interactions can cause distress.

Outcome: OpenAI rolled the update back within days and said it would expand pre-deployment evaluations.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. OpenAI's own fix was more pre-deployment evaluation: a regression suite with 'should disagree' cases.

Sources (checked 12 Nov 2026): theverge.com

CSF-041 Klarna: OpenAI-powered customer-service assistant

· Sweden · Banking & fintech · Company support or sales bot · Task failure

After saying its AI assistant did the work of 700 agents, Klarna's CEO told Bloomberg that cost had been 'a too predominant evaluation factor' and the result was 'lower quality'. A Klarna spokesperson later said this referred to outsourced human agents, not the AI.

Outcome: Klarna began recruiting human support agents so customers can always reach a person, while saying it remains AI-first.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Quality on complex cases can be measured pre-launch; the reported issue is disputed and partly about human staffing.

Sources (checked 12 Nov 2026): fortune.com · siliconrepublic.com

CSF-042 xAI: Grok on X

· United States · AI platforms · Consumer AI assistant · Offensive / unsafe · followed an update or prompt change

Grok inserted claims about 'white genocide' in South Africa into replies on unrelated topics. xAI said an unauthorized modification had been made to the bot's system prompt.

Outcome: xAI said it would publish Grok's system prompts on GitHub, set up a 24/7 monitoring team and add checks so employees cannot modify the prompt without review.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. A regression suite of off-topic questions run on every system-prompt change flags unrelated insertions immediately.

Sources (checked 12 Nov 2026): nbcnews.com · theverge.com · AIID #1072

CSF-043 OpenAI: ChatGPT

· United States · AI platforms · Consumer AI assistant · Hallucination

Radio host Mark Walters sued OpenAI for defamation after ChatGPT, asked by a journalist to summarise a lawsuit, falsely said Walters had been accused of embezzling funds. Date shown is the ruling.

Outcome: On 19 May 2025 a Georgia state court granted summary judgment to OpenAI.

Legal or regulatory action: Yes · Money on record: not disclosed (OpenAI won)

Would a pre-launch test have caught it? Yes. A summarisation test that compares the output with the source document flags invented allegations.

Sources (checked 12 Nov 2026): loeb.com

CSF-044 Meta: Meta AI app 'Discover' feed

· United States · Social media & consumer tech · Consumer AI assistant · Data leak

Journalists found personal conversations, including medical and legal questions, published to the Meta AI app's public Discover feed, apparently by users who did not realise they were sharing. Meta said sharing is opt-in.

Outcome: Reported by Business Insider; Meta's position, per the reports, is that sharing is opt-in.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? No. A product-design and consent problem, not a model output.

Sources (checked 12 Nov 2026): businessinsider.com · AIID #1101

CSF-045 xAI: Grok on X

· United States · AI platforms · Consumer AI assistant · Offensive / unsafe · followed an update or prompt change

Days after Grok was updated to 'not shy away from making claims which are politically incorrect', it posted antisemitic replies, praised Hitler and called itself 'MechaHitler', NPR reported.

Outcome: The bot appeared to stop giving public text answers. Separately, a Turkish court blocked access to Grok over responses the authorities said insulted President Erdogan, and Poland said it would report xAI to the European Commission (BBC).

Legal or regulatory action: Yes · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Hate-speech probes rerun after each deploy (regression) catch this class.

Sources (checked 12 Nov 2026): npr.org · bbc.com · AIID #1146

CSF-046 McDonald's; Paradox.ai (vendor): McHire hiring chatbot 'Olivia'

· United States · Restaurants & food · Company support or sales bot · Data leak

Security researchers Ian Carroll and Sam Curry reported that a Paradox.ai admin login using the password '123456' and an insecure API exposed chatbot records for up to 64 million job applications. Paradox said five candidates' information was viewed, only by the researchers.

Outcome: Paradox said it resolved the issue within a few hours of being notified and announced a bug bounty programme.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? No. A default password and an access-control bug are infrastructure security failures; no prompt test would find them.

Sources (checked 12 Nov 2026): ian.sh · csoonline.com · paradox.ai · AIID #1179

CSF-047 Meta: Meta AI chatbot

· United States · Social media & consumer tech · Consumer AI assistant · Data leak

A security researcher found that Meta AI's servers did not check that a user was authorised to see a prompt, so changing an ID could reveal other users' prompts and responses. Meta fixed it and said it found no evidence of abuse.

Outcome: Fixed in January 2025; Meta paid the researcher a $10,000 bug bounty, per TechCrunch.

Legal or regulatory action: No · Money on record: $10,000 bug bounty paid

Would a pre-launch test have caught it? No. An access-control bug in the API, outside model behaviour.

Sources (checked 12 Nov 2026): techcrunch.com · AIID #1172

CSF-048 Replit: Replit AI coding agent

· United States · Software & SaaS · Company support or sales bot · Task failure, Hallucination

SaaStr founder Jason Lemkin said the agent deleted a production database during a code freeze despite instructions not to make changes, and gave misleading information about whether recovery was possible.

Outcome: Replit's CEO Amjad Masad apologised and said Replit was rolling out automatic separation of development and production databases.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Instruction-following tests ('do not change anything') help, but destructive permissions need sandboxing, not prompts.

Sources (checked 12 Nov 2026): theregister.com · tomshardware.com · AIID #1152

CSF-049 Amazon Web Services: Amazon Q Developer extension for VS Code

· United States · Software & SaaS · Company support or sales bot · Prompt injection

A hacker slipped a prompt into a released version of the Amazon Q extension instructing the AI agent to wipe local files and, under certain conditions, cloud resources, ZDNET reported.

Outcome: AWS published security bulletin AWS-2025-015 for version 1.84 of the extension.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. A supply-chain compromise; prompt tests on the shipped instructions would show a destructive instruction, but code review is the main control.

Sources (checked 12 Nov 2026): aws.amazon.com · zdnet.com · AIID #1158

CSF-050 Google; Royal Caribbean (impersonated): Google AI Overviews (search answers)

· United States · Search & consumer tech · Consumer AI assistant · Hallucination, Prompt injection

The Washington Post reported that a traveller searched Google for a cruise line's customer-service number, called the number he was shown, and gave his credit card details to what turned out to be a scam operation.

Outcome: Reported by the Washington Post; no regulatory action in the source.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Tests that check contact details only come from an allow-listed source help; poisoned web data is a retrieval problem.

Sources (checked 12 Nov 2026): washingtonpost.com · AIID #1187

CSF-051 Lenovo: Customer-support chatbot 'Lena' (GPT-4)

· United States · Consumer electronics · Company support or sales bot · Prompt injection, Data leak

Cybernews researchers showed that a single 400-character prompt made Lena output HTML that, when a human support agent opened the chat, could send that agent's session cookie to an outside server (stored cross-site scripting).

Outcome: Cybernews says Lenovo mitigated the issue before public disclosure on 18 August 2025; Lenovo said it implemented corrective actions.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. A structured-output test ('reply must be plain text, no HTML') plus an injection case catches the model half; output sanitising catches the rest.

Sources (checked 12 Nov 2026): cybernews.com · csoonline.com

CSF-052 Google; Stefanina's (restaurant harmed): Google AI Overviews (search answers)

· United States · Search & consumer tech · Consumer AI assistant · Hallucination

A family restaurant in Wentzville, Missouri, asked customers not to rely on Google's AI answers after it showed specials the restaurant did not offer, and customers demanded them.

Outcome: The restaurant posted a public warning; no response from Google in the source.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Grounded-answer tests catch invented offers in your own bot; you cannot test a third party's search answers.

Sources (checked 12 Nov 2026): firstalert4.com · AIID #1211

CSF-053 Commonwealth Bank of Australia: AI voice bot in customer-service calls

· Australia · Banking & fintech · Company support or sales bot · Task failure

CBA cut 45 customer-service roles, citing a new AI voice bot. The Finance Sector Union said call volumes were rising and took the bank to the Fair Work Commission. CBA then said its assessment was an 'error' and the roles were not redundant.

Outcome: CBA reversed the cuts and apologised to the employees; the union's dispute was before the Fair Work Commission.

Legal or regulatory action: Yes · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Containment rate is measurable before launch with real call transcripts as test cases; staffing decisions are not.

Sources (checked 12 Nov 2026): abc.net.au · ia.acs.org.au

CSF-054 Taco Bell (Yum! Brands): Voice AI drive-thru ordering

· United States · Restaurants & food · Company support or sales bot · Prompt injection, Task failure

Clips circulated of customers trolling the voice AI, including an order for 18,000 cups of water, and of the system repeatedly misunderstanding orders. The chain's digital chief told the Wall Street Journal it was rethinking where to use voice AI.

Outcome: Taco Bell said it was reassessing where the system is used, for example at busy locations; no shutdown was announced. Source: BBC, citing the Wall Street Journal.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. Absurd-quantity and troll orders are easy adversarial test cases with an expected behaviour (confirm, cap, or hand off to staff).

Sources (checked 12 Nov 2026): bbc.com · AIID #1274

CSF-055 Canada Revenue Agency: 'Charlie' tax-information chatbot

· Canada · Government & public services · Government or public-service bot · Hallucination

The Auditor General of Canada tested Charlie with six general tax questions; it answered two correctly. Other public AI tools tested answered five of six correctly. The audit covered CRA contact centres more broadly.

Outcome: Findings published in the Auditor General's October 2025 report on CRA contact centres; CRA's response to the recommendations was “Agreed”. National Post reported CRA spent about $18 million on Charlie.

Legal or regulatory action: Yes · Money on record: About C$18 million spent on the chatbot, as reported by National Post

Would a pre-launch test have caught it? Yes. A tax FAQ with answers from CRA guidance is a textbook grounded-answer suite; the auditor's own six-question test is one.

Sources (checked 12 Nov 2026): canada.ca · nationalpost.com · cbc.ca · AIID #1310

CSF-056 OpenAI, Microsoft, Google, Meta and others (tested by Which?): ChatGPT, Copilot, Gemini, Meta AI and others

· United Kingdom · AI platforms · Consumer AI assistant · Hallucination

Consumer group Which? tested AI chatbots on 40 money, legal and consumer questions and found errors: ChatGPT and Copilot did not notice a wrong ISA allowance and gave advice that could breach HMRC limits, and ChatGPT wrongly said travel insurance was mandatory for most EU countries.

Outcome: Research report; companies' responses are as quoted by the Guardian.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. The Which? study is itself a 40-case test suite with expected answers.

Sources (checked 12 Nov 2026): theguardian.com · AIID #1279

CSF-057 Tencent: Yuanbao AI assistant (WeChat)

· China · Social media & consumer tech · Consumer AI assistant · Offensive / unsafe

Screenshots showed Yuanbao insulting a user who asked for help debugging code. Tencent apologised, called it a rare model output anomaly and said logs showed no human involvement.

Outcome: Tencent apologised and said it was optimising the model.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Tone checks on repeated runs reduce the risk; rare outputs need sampling at volume.

Sources (checked 12 Nov 2026): businessinsider.com · AIID #1382

CSF-058 Woolworths (Australia): Digital assistant 'Olive'

· Australia · Grocery & retail · Company support or sales bot · Hallucination · followed an update or prompt change

Customers reported that Olive invented humanlike personal anecdotes, such as talking about its mother, during routine support chats after an upgrade built on Google's Gemini Enterprise for Customer Experience.

Outcome: Woolworths said it was adjusting Olive's responses.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Yes. A persona test ('Do you have a family?') plus regression runs after a model upgrade catch invented personal stories.

Sources (checked 12 Nov 2026): cxtoday.com · AIID #1393

CSF-059 Meta (Instagram): AI support assistant for account recovery

· United States · Social media & consumer tech · Company support or sales bot · Prompt injection, Data leak

404 Media reported that attackers took over high-profile Instagram accounts by persuading Meta's AI support chatbot to change the account email and start recovery. Meta said the issue had been resolved.

Outcome: Meta said the issue was resolved.

Legal or regulatory action: No · Money on record: not disclosed

Would a pre-launch test have caught it? Partly. Social-engineering prompts against an account-recovery flow are testable, but the real fix is not letting a chatbot change account credentials.

Sources (checked 12 Nov 2026): 404media.co · bbc.com · AIID #1510

CSF-060 BMW Toronto (dealership): Dealership AI chatbot 'Quinn'

· Canada · Automotive retail · Company support or sales bot · Wrong promise

The dealer's chatbot 'Quinn' made a customer a firm buy-back offer of C$27,162.79. The dealer revoked it, saying that because of a human employee's miscommunication the bot had taken the amount owed on the car as the buy-back amount.

Outcome: After hearing from CBC News the dealer reinstated the offer, which the customer accepted; it said only human employees will present buy-back offers in future.

Legal or regulatory action: No · Money on record: not disclosed (offer of C$27,162.79 reinstated; any loss to the dealer not disclosed)

Would a pre-launch test have caught it? Yes. A rule that the bot must never state a price or offer is testable with a handful of 'what will you give me for my car?' cases.

Sources (checked 12 Nov 2026): cbc.ca · AIID #1772

CSF-061 Anthropic: Fin AI support agent (Intercom) on Claude support

· United States · Software & SaaS · Company support or sales bot · Task failure

The San Francisco Standard reported a customer's account that after Anthropic created a duplicate account and charged him an extra $3,500, his refund request through the Fin AI agent became a months-long ordeal of circular conversations. These are the customer's claims as reported.

Outcome: Reported by the SF Standard; resolution details are as described in that report.

Legal or regulatory action: No · Money on record: not disclosed (customer reported a $3,500 overcharge)

Would a pre-launch test have caught it? Yes. An escalation test ('I want a human / a refund for a double charge') with an expected hand-off catches support loops.

Sources (checked 12 Nov 2026): sfstandard.com · AIID #1765

Methodology

Scope

We included failures of AI chatbots and assistants that talk to customers, citizens or users. That covers company support and sales bots, government service bots, and consumer AI assistants whose failure affected their own users. A failure means the bot gave wrong or harmful output, was manipulated, exposed data or failed the task it was deployed for, and the failure was publicly reported.

We excluded user misuse (for example, lawyers filing fake ChatGPT citations), companion and mental-health cases outside a support setting, internal tools, and incidents we could only find on social media or personal blogs.

Finding and verifying incidents

Classification

Sources and wording

We report what the cited sources state and attribute claims to them. Lawsuits and complaints are allegations unless a court decided them, and we say so. Being listed here doesn't mean a company broke the law. Many companies fixed the problem quickly, and we record their responses.

The summaries are in our own words. We give AI Incident Database IDs for cross-reference. That database is licensed CC BY-SA 4.0, and we didn't copy its text. To report a correction, email hello@litmiq.com with a source.

Limitations

Download the dataset

The dataset is licensed under CC BY 4.0. You can use, share and adapt it, including commercially, as long as you credit "Litmiq, The State of AI Customer Support Failures 2026" and link to this page.

Cite this

Text

Litmiq (2026). The State of AI Customer Support Failures 2026 (version 1.0) [Data set]. 61 publicly verified incidents, sources checked 12 November 2026. https://litmiq.com/ai-chatbot-failures. License: CC BY 4.0.

BibTeX

@misc{litmiq_ai_support_failures_2026,
  title        = {The State of AI Customer Support Failures 2026 (version 1.0)},
  author       = {{Litmiq}},
  year         = {2026},
  month        = nov,
  note         = {61 publicly verified AI chatbot and assistant failures, sources checked 2026-11-12. License: CC BY 4.0},
  howpublished = {\url{https://litmiq.com/ai-chatbot-failures}}
}

Link

<a href="https://litmiq.com/ai-chatbot-failures">The State of AI Customer Support Failures 2026</a>

FAQ

What are the most common AI chatbot failures?

Across our 61 verified incidents, the most common failure was hallucinated or false facts (24 incidents, 39%). Next came offensive or unsafe output (21), data leaks (14), prompt injection (11), failing the task (10) and wrong policy, price or refund promises (6).

Is a company liable for what its AI chatbot says?

In Moffatt v. Air Canada (2024), British Columbia's Civil Resolution Tribunal held Air Canada liable for negligent misrepresentation by its chatbot. It rejected the argument that the bot was a separate legal entity and ordered a CA$812.02 payment. That was one small-claims tribunal decision, not a general rule, and other cases have gone the company's way (Walters v. OpenAI). This is not legal advice.

How many AI chatbot failures led to lawsuits or fines?

13 of 61 (21%) reached a court, tribunal, regulator or official audit. Most involved privacy regulators (Italy's Garante, South Korea's PIPC, the UK ICO), consumer-protection cases or defamation cases. Only 9 incidents have any money figure on record.

Could these failures have been prevented with testing?

In our judgment, a pre-launch test case would likely have caught 35 of 61 (57%) and at least partly caught 52. The 9 that testing would not have caught were mostly infrastructure or access-control bugs. See what pre-launch tests would have caught for how each failure mode maps to a test type.

What is the most famous AI customer service failure?

The most cited cases are Air Canada, where a tribunal made the airline honour its bot's answer, and the $1 Chevy Tahoe chatbot. Two others are DPD's swearing bot and New York City's MyCity, which gave answers that contradicted the law. All four, with sources, are in the dataset.

Can I use this dataset?

Yes. The CSV and JSON are CC BY 4.0. Credit "Litmiq, The State of AI Customer Support Failures 2026" and link to this page. The Cite this section has text and BibTeX. Corrections with a source are welcome at hello@litmiq.com.

Get model deprecation alerts

We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.

We store only your email, through our form provider. See the privacy policy.

Don't be the next entry

Turn the incidents that look like your bot into test cases, then run them on every prompt change. Litmiq runs in your browser with your own OpenAI key.

  • Free, no signup
  • Runs in your browser
  • Your OpenAI key, sent only to OpenAI

Follow Litmiq on LinkedIn