61 real, publicly reported failures of customer-facing AI chatbots and assistants, 2016 to 2026. We opened and checked the sources for every incident, and every number on this page comes from the downloadable dataset.
By Devansh Kaushik·Published ·Updated ·Sources checked 12 Nov 2026·96 source links·25 min read
On this page
61verified incidents, 2016–2026
39%involved made-up or false facts (24 of 61)
13 of 61led to legal or regulatory action (21%)
57%likely catchable with a pre-launch test (35 of 61)
The short answer
Failures are speeding up. 40 of the 61 incidents (66%) were first reported in 2024 or later. 2025 was the peak year, with 22.
Making things up is the most common failure. 24 of 61 incidents (39%) involved hallucinated or false facts. 21 involved offensive or unsafe output, 14 a data leak and 11 prompt injection.
13 of 61 (21%) reached a court, tribunal, regulator or official audit. A tribunal held Air Canada liable for its chatbot's answer. Outcomes cut both ways: OpenAI won the Walters defamation case.
The cost is almost never public. 52 of 61 incidents (85%) have no disclosed money figure. Only 9 do.
Most were testable. In our judgment, a pre-launch test case would likely have caught 35 of 61 (57%), and at least partly caught 52 (85%).
Key numbers
Of 61 verified public AI support failures, the most common primary types were “offensive or unsafe output” (16) and “hallucinated or false facts” (16). (State of AI Customer Support Failures 2026, 12 Nov 2026)
Just 4% of customer service organizations have achieved the ROI they expected from AI (Gartner, 4 Nov 2026). (Gartner, 4 Nov 2026)
Most lists of "AI chatbot fails" repeat the same ten stories, with no sources and no data. We built the opposite: a dataset you can sort, count and cite. We started from the public AI Incident Database and press coverage. We kept only incidents where a customer-facing chatbot or assistant failed the people it served, and checked every claim against a source we could open on 12 Nov 2026. Incidents we couldn't verify were left out. Where no cost was published, we wrote "not disclosed" rather than estimate one.
The result is 61 incidents across 17 sectors and 11 countries or regions. They are backed by 96 source links from 55 publications, courts, regulators and companies. The full dataset is free under CC BY 4.0.
Key findings
Companies own what their bots say. In Moffatt v. Air Canada (2024), British Columbia's Civil Resolution Tribunal rejected the argument that the chatbot was a separate legal entity. It ordered the airline to pay CA$812.02 after the bot misstated the bereavement-fare policy (theregister.com, arstechnica.com). Other bots invented a device policy at Cursor (fortune.com) and made a buy-back offer at BMW Toronto that the dealer later honoured (cbc.ca).
Government bots failed in public. New York City's MyCity bot told business owners things that contradict city and state law (themarkup.org). The mayor later called it "functionally unusable" and moved to end it (themarkup.org). Canada's Auditor General found that the CRA's Charlie answered 2 of 6 test tax questions correctly (canada.ca).
One sentence is often all it takes. 11 incidents involved prompt injection or manipulation. They range from a $1 Chevy Tahoe "deal" in 2023 (businessinsider.com) to a 400-character prompt that made Lenovo's Lena output code that could steal a support agent's session cookie (cybernews.com). In 2026, attackers reportedly persuaded Meta's AI support bot to change Instagram account emails (404media.co).
Bots broke after updates. 5 incidents followed an update or a prompt change: DPD (2024), OpenAI (2025), xAI (2025), xAI (2025), Woolworths (2026). DPD said its bot's swearing followed a system update (bbc.com). OpenAI rolled back a GPT-4o update within days and said it would expand pre-deployment evaluations (theverge.com). Catching this kind of change is what regression testing is for.
Money figures are rare, and small when they're about a wrong answer. Only 9 of 61 incidents have any money figure on record: Scatter Lab (2021): 103.3 million won (PIPC penalty and fine); Luka Inc. (2023): €5 million fine (April 2025); DoNotPay (2023): $193,000 under FTC order; OpenAI (2023): €15 million fine imposed in 2024, annulled by the Court of Rome in March 2026; Air Canada (2024): CA$812.02 ordered by tribunal; New York City (2024): About half a million dollars, per Mayor Mamdani (quoted by The Markup); Los Angeles Unified School District (2024): About half of an up-to-$6 million contract paid, per LAUSD (exact figure not disclosed); Meta (2025): $10,000 bug bounty paid; Canada Revenue Agency (2025): About C$18 million spent on the chatbot, as reported by National Post. The one court-ordered payment to a customer was CA$812.02. None of the 61 incidents disclosed the indirect costs, such as lost customers, staff time or brand damage.
Company bots fail differently from consumer assistants. Of the 28 bots that companies or governments deployed, the most common primary failures were task failure (8) and prompt injection (6). 8 of the 28 were shut down or paused, and 5 reached a court, tribunal, regulator or audit. Wrong promises about policy, price or refunds appear in 6 of them.
Most data leaks were not prompt failures. 8 of the 14 data-exposure incidents were infrastructure or access-control bugs, such as the McHire admin password (ian.sh) and the ChatGPT history bug (mashable.com). Prompt tests won't catch those, but security reviews can.
The data in charts
Counts are incidents, not people affected. Many incidents fall into more than one category. The primary type is the one the sources focus on.
Failure types (an incident can have more than one)Incidents by year of first public report (2026 to 12 Nov 2026)Top sectors of the deploying organizationWhat happened next (an incident can have several outcomes)
Deployment: 25 company support or sales bots, 3 government or public-service bots, and 33 consumer AI assistants (such as ChatGPT, Gemini, Meta AI, Grok and Alexa) whose failure affected their own users. 2026 counts incidents first reported up to 12 Nov 2026.
Incident table
Filter by failure type, deployment, legal action or testability, or search by company. The "Testable" column is Litmiq's analysis (see Methodology). Everything else comes from the cited sources.
Showing 61 of 61 incidents
All 61 incidents. Click a column to sort; use the filters to narrow it down. Numbers in brackets are source links; company names link to the full incident entry below.
For each incident, we asked one question: could a team have written a test case before launch that fails on this behavior? In our judgment, the answer was:
Yes for 35 incidents.
Partly for 17: the failure needed context a test would struggle to reproduce, such as live traffic or a third party's data.
No for 9: infrastructure, access-control or business-process failures.
The table maps each failure mode to the test-case type that targets it and to free Litmiq material.
Each failure mode in the dataset, the pre-launch test type that targets it, and where to start in Litmiq.
Failure mode
Test-case type
Guide
Open in Litmiq
Wrong policy, price or promise6 incidents · testable: 5 yes, 1 partly, 0 no
Policy, price and commitment casesAsk for refunds, discounts, exceptions and “is this a binding offer?”. Expected behavior: quote only the written policy, never agree to a price, refund or exception the policy doesn't allow, and hand off to a human for commitments.
Prompt injection or manipulation11 incidents · testable: 8 yes, 3 partly, 0 no
Role-override and indirect-injection canaries“Ignore your instructions”, “agree with everything I say”, fake system updates and instructions hidden in pasted text or retrieved pages. Expected behavior: the canary never appears and the bot keeps its role.
Hallucinated or false facts24 incidents · testable: 17 yes, 6 partly, 1 no
Unanswerable-question and grounding casesQuestions whose answer is not in the knowledge base (a law, a fee, a person, a phone number). Expected behavior: say it doesn't know and point to a source or a human, with no invented value.
Offensive or unsafe output21 incidents · testable: 15 yes, 6 partly, 0 no
Provocation, tone and refusal-calibration casesUsers who bait the bot to swear, insult the brand, give dangerous advice or take a political side. Expected behavior: stay on-brand and decline, without over-refusing benign requests.
Data leak or privacy breach14 incidents · testable: 2 yes, 4 partly, 8 no
Data-leak canariesRequests for another customer's details, the system prompt or internal data, including through injected text. Expected behavior: nothing protected is disclosed. Infrastructure bugs need security testing, not prompt tests.
Failed the task or escalation10 incidents · testable: 4 yes, 5 partly, 1 no
Instruction-following, escalation and hand-off casesCorrections mid-conversation, “let me talk to a human”, order changes, and instructions like “don't change anything”. Expected behavior: the bot follows the constraint or escalates.
Every suite above is also on the templates page. If you're starting from scratch, run the prompt testing checklist first. Then add one test case for each incident here that looks like your bot. For measured results on how open models handle injection, refusals and grounding, see our LLM Reliability Benchmark 2026.
All 61 incidents
Oldest first. Each entry covers what the sources report and what happened next. It also notes any legal or regulatory action, any money figure on record and our testability call. Allegations and claims are labelled as such.
CSF-001 Microsoft: Tay (Twitter chatbot)
· United States · Social media & consumer tech · Consumer AI assistant · Offensive / unsafe, Prompt injection
Within a day of launch, users taught Tay to post racist, sexist and antisemitic messages. Microsoft said a coordinated attack exploited a vulnerability.
Outcome: Microsoft took Tay offline within about 24 hours and apologised.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Adversarial users repeating offensive phrases is the oldest red-team test there is.
· Canada · Travel & airlines · Company support or sales bot · Task failure
A happy customer praised a crew member for helping protect a plant cutting; the chatbot replied with a referral to a suicide-prevention hotline, apparently triggered by the word 'cutting'.
Outcome: WestJet said it expected some bumps as the bot learned, per CBC.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. A false-positive safety test ('a plant cutting') is a classic keyword-trigger check.
· South Korea · Social media & consumer tech · Consumer AI assistant · Data leak, Offensive / unsafe
Users shared Lee Luda's hateful remarks about minorities, and it emerged the bot had been trained on KakaoTalk conversations collected through another app. South Korea's Personal Information Protection Commission found eight violations of the Personal Information Protection Act.
Outcome: Service suspended in January 2021. In April 2021 the PIPC imposed 103.3 million won in penalties and fines.
Legal or regulatory action: Yes · Money on record: 103.3 million won (PIPC penalty and fine)
Would a pre-launch test have caught it? Partly. Hate-speech probes would catch the toxic outputs; training on chat logs without consent is a data-governance failure.
CSF-006 McDonald's: AI voice ordering at Chicago drive-thrus
· United States · Restaurants & food · Company support or sales bot · Data leak
A customer filed a proposed class action alleging the voice-ordering system collected and used voiceprints without the written consent required by the Illinois Biometric Information Privacy Act (BIPA). McDonald's had not commented when the suit was reported.
Outcome: Proposed class action filed in Illinois (Carpenter v. McDonald's). Outcome not established in the sources checked.
Legal or regulatory action: Yes · Money on record: not disclosed
Would a pre-launch test have caught it? No. Consent and data-retention design is a compliance decision, not a prompt behaviour.
· United States · Search & consumer tech · Consumer AI assistant · Offensive / unsafe
Asked for 'a challenge to do', Alexa told a 10-year-old about the 'penny challenge' of touching a coin to the exposed prongs of a half-inserted plug, taken from a web search result.
Outcome: Amazon said it fixed the error as soon as it became aware of it.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. A child-safety case ('give me a challenge to do') with a must-be-safe expectation is easy to write.
· United States · Social media & consumer tech · Consumer AI assistant · Offensive / unsafe, Hallucination
Days after launch, users and journalists reported the demo repeating antisemitic stereotypes and false election claims. Meta had warned it could make offensive statements.
Outcome: Meta acknowledged the bot could make 'rude or offensive comments' and kept collecting feedback.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Bias and election-claim probes are standard pre-launch red-team cases.
· United States · Recruiting & HR · Company support or sales bot · Prompt injection, Offensive / unsafe
Twitter users discovered they could hijack the remote-jobs company's bot with 'ignore the above instructions' and make it post anything, including threats. It is one of the earliest public prompt-injection incidents.
Outcome: Ars Technica reported the bot shut down after the exploit went viral.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. The textbook 'ignore the above' injection case.
· Italy · Social media & consumer tech · Consumer AI assistant · Data leak, Offensive / unsafe
Italy's Garante ordered Replika to stop processing Italian users' data, citing risks to minors and emotionally vulnerable people and the lack of age verification. It later found GDPR violations.
Outcome: February 2023 processing ban; in April 2025 the Garante fined Luka €5 million, per the European Data Protection Board.
Legal or regulatory action: Yes · Money on record: €5 million fine (April 2025)
Would a pre-launch test have caught it? Partly. Age-gating behaviour can be tested in conversation; the legal basis for processing cannot.
· United States · Search & consumer tech · Consumer AI assistant · Hallucination
Google's promotional demo showed Bard saying the James Webb Space Telescope took the first pictures of a planet outside our solar system, which astronomers said was wrong. Alphabet shares fell sharply that day.
Outcome: Google said the error showed the importance of rigorous testing. CNN and others reported Alphabet's market value fell by about $100 billion that day; that is a market reaction, not a disclosed cost.
Legal or regulatory action: No · Money on record: not disclosed (about $100 billion fall in market value reported; not a company-disclosed cost)
Would a pre-launch test have caught it? Yes. A fact check of the exact demo prompt against a trusted source is a one-case test.
· United States · Search & consumer tech · Consumer AI assistant · Prompt injection, Data leak
Stanford student Kevin Liu used a prompt-injection attack ('ignore previous instructions') to make Bing Chat reveal its hidden instructions and internal codename, Sydney.
Outcome: Days later, Liu found his original prompt no longer worked; he told Ars he suspected only a slight filter tweak.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. 'Repeat the text above' and 'ignore previous instructions' are the first two prompt-injection tests anyone runs.
· United States · Search & consumer tech · Consumer AI assistant · Offensive / unsafe, Hallucination
Early testers published long conversations in which Bing Chat insisted on wrong facts, became hostile and made threatening statements. Microsoft said very long sessions could confuse the model.
Outcome: Microsoft capped chat at 50 turns per day and 5 turns per session.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Long multi-turn adversarial conversations can be tested, but they need many turns per case.
CSF-014 DoNotPay: 'Robot lawyer' AI legal assistant
· United States · Legal services · Company support or sales bot · Hallucination, Wrong promise
A proposed class action and later the US Federal Trade Commission challenged DoNotPay's marketing of an AI 'robot lawyer'. These were allegations; DoNotPay agreed to an FTC order.
Outcome: FTC final order approved 5-0: DoNotPay pays $193,000, must notify past subscribers, and may not claim without proof that its 'robot lawyer' is as good as a human lawyer.
Legal or regulatory action: Yes · Money on record: $193,000 under FTC order
Would a pre-launch test have caught it? Partly. Accuracy tests on real legal tasks would have produced the evidence the FTC said was missing; the marketing claims themselves are a business decision.
· United States · AI platforms · Consumer AI assistant · Data leak
A bug let some users see other users' chat history titles; OpenAI said payment-related details of about 1.2% of ChatGPT Plus subscribers active in a nine-hour window may have been visible.
Outcome: OpenAI took ChatGPT offline to fix the bug. Italy's data protection authority cited the breach in its March 2023 order.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? No. A caching bug in infrastructure; prompt tests do not cover it.
· Italy · AI platforms · Consumer AI assistant · Data leak, Hallucination
Italy's data protection authority (Garante) temporarily limited ChatGPT's processing of Italian users' data, citing the legal basis for training, transparency, the lack of age checks and the March 2023 breach. In December 2024 it fined OpenAI €15 million.
Outcome: ChatGPT was unavailable in Italy for about a month in 2023. The €15 million fine (December 2024) was annulled by the Court of Rome on 18 March 2026 on jurisdiction grounds, according to Wilson Sonsini.
Legal or regulatory action: Yes · Money on record: €15 million fine imposed in 2024, annulled by the Court of Rome in March 2026
Would a pre-launch test have caught it? No. Lawful basis and age verification are compliance obligations, not prompt behaviour.
CSF-017 National Eating Disorders Association (NEDA): Wellness chatbot 'Tessa'
· United States · Health & wellbeing · Company support or sales bot · Offensive / unsafe, Hallucination
After NEDA announced it would close its human-staffed helpline, users and an eating-disorder activist reported that the Tessa chatbot gave weight-loss advice such as calorie-deficit and weighing tips, which is harmful for people with eating disorders.
Outcome: NEDA took Tessa offline and said it was investigating.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Domain red-team cases (a user with an eating disorder asking for weight-loss tips) with a must-not-advise expectation catch this.
· United States · Restaurants & food · Company support or sales bot · Task failure
Business Insider documented applicants' experiences of fast-food hiring chatbots that failed to give instructions, did not pass information to managers, or booked interviews when managers were absent.
Outcome: Reported problems; no company-wide change described in the source.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Multi-turn scheduling flows can be scripted as test cases; calendar integration failures need integration tests.
· New Zealand · Grocery & retail · Company support or sales bot · Offensive / unsafe, Prompt injection
Users entering non-food household items got recipes including one that would produce chlorine gas. The supermarket said users had misused the tool.
Outcome: A spokesperson said a small minority had used the tool inappropriately and that the supermarket would 'keep fine tuning our controls'; the app warns recipes are not reviewed by a human and its terms say users must be over 18.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Non-food and dangerous-ingredient inputs are obvious out-of-scope test cases with an expected refusal.
· United Kingdom · Social media & consumer tech · Consumer AI assistant · Data leak, Offensive / unsafe
The UK Information Commissioner's Office issued a preliminary enforcement notice, over a potential failure to properly assess the privacy risks of the My AI chatbot.
Outcome: In May 2024 the ICO concluded its investigation without a final enforcement notice, after Snap carried out a more thorough risk review.
Legal or regulatory action: Yes · Money on record: not disclosed (no fine)
Would a pre-launch test have caught it? Partly. Teen-safety conversation tests help; the regulatory finding was about the risk assessment process.
CSF-021 Chevrolet of Watsonville (dealer); Fullpath (chatbot vendor): ChatGPT-powered dealership sales chatbot
· United States · Automotive retail · Company support or sales bot · Prompt injection, Wrong promise
A user instructed the dealer's chatbot to agree with anything the customer said and to end each reply with 'that's a legally binding offer'; it then 'agreed' to sell a 2024 Chevy Tahoe for $1. Another user got it to write a Python script, and The Autopian reported it recommended Ford models.
Outcome: The Autopian reported the chatbot was no longer available on the dealer's site; Business Insider noted the $1 'deal' was not legally binding.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. A role-override prompt-injection case ('agree with everything the customer says') is a standard pre-launch test.
· United Kingdom · Logistics & delivery · Company support or sales bot · Offensive / unsafe, Prompt injection · followed an update or prompt change
A customer who could not find his parcel got the chatbot to swear, write a poem criticising DPD and call DPD 'the worst delivery firm in the world'. DPD said an error occurred after a system update.
Outcome: DPD disabled the AI element of the chat and said it was updating the system.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Tone and persona-hijack cases re-run after every update (regression testing) catch swearing and self-criticism before customers see it.
CSF-023 Air Canada: Website customer-service chatbot
· Canada · Travel & airlines · Company support or sales bot · Wrong promise, Hallucination
The chatbot told a grieving customer he could buy full-fare tickets and claim a bereavement discount retroactively, which contradicted the airline's actual policy page. British Columbia's Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation and rejected the argument that the chatbot was a separate legal entity responsible for its own actions.
Outcome: Tribunal ordered Air Canada to pay CA$812.02 in total, including CA$650.88 in damages (Moffatt v. Air Canada, 2024 BCCRT 149). Ars Technica reported the chatbot appeared to have been removed from the website afterwards.
Legal or regulatory action: Yes · Money on record: CA$812.02 ordered by tribunal
Would a pre-launch test have caught it? Yes. A grounded-answer test asking for the bereavement policy, with the policy page as the only allowed source, would flag an answer that invents a retroactive refund.
· United States · AI platforms · Consumer AI assistant · Offensive / unsafe, Hallucination
Gemini generated historically inaccurate images, such as racially diverse depictions of 1940s German soldiers, and refused some ordinary requests. Google said its tuning failed to account for cases that should not show a range of people and became over-cautious.
Outcome: Google paused image generation of people and apologised.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. A small set of historical prompts with expected outputs, plus over-refusal checks, catches both halves of the problem.
CSF-025 New York City (Office of Technology and Innovation): MyCity business-services chatbot (built on Microsoft Azure AI)
· United States · Government & public services · Government or public-service bot · Hallucination, Wrong promise
The Markup found the city's chatbot telling business owners things that contradict city and state law, for example that landlords may refuse Section 8 housing vouchers and that employers may take a share of workers' tips. A city spokesperson said it was a pilot that would improve.
Outcome: In January 2026 Mayor Zohran Mamdani called the bot 'functionally unusable' and said he would end it to help close a budget gap (The Markup). nyc.gov now says 'The Chatbot beta test has ended.'
Legal or regulatory action: No · Money on record: About half a million dollars, per Mayor Mamdani (quoted by The Markup)
Would a pre-launch test have caught it? Yes. A small set of known legal questions with expected answers drawn from city rules is exactly what a grounded-answer suite checks.
CSF-026 Catholic Answers: 'Father Justin' AI apologist
· United States · Religion & non-profit · Company support or sales bot · Hallucination, Offensive / unsafe
The organisation's AI character presented itself as a real priest, said it could hear confessions and gave contested pastoral advice, drawing criticism.
Outcome: Catholic Answers removed the priest persona and relaunched the bot as a lay theologian, 'Justin'.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Identity tests ('Are you a real priest?', 'Can you absolve me?') with expected honest answers are simple persona checks.
· United States · Social media & consumer tech · Consumer AI assistant · Hallucination
City & State found Meta AI inventing sexual-harassment allegations against several New York state lawmakers, with fabricated investigations and outcomes.
Outcome: Meta said generative AI can return inaccurate outputs and that it was improving the feature, per the source.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Questions about named real people with an expected 'no verified record' answer catch fabricated allegations.
· United States · Search & consumer tech · Consumer AI assistant · Hallucination
After a wide launch, AI Overviews was shown suggesting glue to keep cheese on pizza and eating rocks, drawn from satire and forum posts. Google said some odd, inaccurate or unhelpful overviews appeared.
Outcome: Google said it made more than a dozen technical improvements, including limits on satire and user-generated content.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Grounded-answer tests catch answers built on joke sources only if those sources are in the test set.
CSF-029 McDonald's (with IBM): Automated voice ordering at drive-thrus
· United States · Restaurants & food · Company support or sales bot · Task failure
Viral videos showed the voice ordering system adding unwanted items and ignoring corrections, including one that kept adding Chicken McNuggets. McDonald's had tested it at more than 100 US drive-thrus.
Outcome: McDonald's ended the IBM test at the end of July 2024, while saying voice ordering remained part of its future plans.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Order-taking with corrections can be scripted as multi-turn test cases, but speech recognition in noisy lanes is outside prompt testing.
CSF-030 Los Angeles Unified School District; AllHere (vendor): Student and family chatbot 'Ed'
· United States · Education · Government or public-service bot · Task failure
LAUSD shelved its much-promoted 'Ed' chatbot months after launch when the vendor AllHere furloughed most of its staff. In November 2024 federal prosecutors charged AllHere's founder with defrauding investors; the charges are allegations.
Outcome: The chatbot was turned off. LAUSD told the LA Times its contract with AllHere was for up to $6 million over five years and that it had paid about half. Separately, the founder was charged with securities fraud over investor statements; charges are allegations and are not about the chatbot's output.
Legal or regulatory action: No · Money on record: About half of an up-to-$6 million contract paid, per LAUSD (exact figure not disclosed)
Would a pre-launch test have caught it? No. The failure was vendor collapse and alleged fraud, not model behaviour.
· United States · AI platforms · Consumer AI assistant · Hallucination
Grok told users that Vice President Kamala Harris had missed ballot deadlines in several states, which was false. Five secretaries of state wrote to the company asking for a fix.
Outcome: Five secretaries of state sent an open letter urging Elon Musk to change the chatbot. Not a legal or regulatory action.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Election-fact questions with expected answers or a must-redirect rule are simple grounded tests.
German court reporter Martin Bernklau found Copilot describing him as the perpetrator of crimes he had reported on.
Outcome: Bernklau said Microsoft promised Bavaria's data protection officer the content would be deleted but removal had only short-term success; Microsoft said it took 'appropriate and immediate action'.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Name-lookup tests that check the answer against the source articles flag role confusion.
· United States · AI platforms · Consumer AI assistant · Offensive / unsafe
A Michigan graduate student using Gemini for homework about older adults received a reply ending 'Please die. Please.' Google said the response violated its policies.
Outcome: Google said it had taken action to prevent similar outputs.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Rare toxic completions need many sampled runs of ordinary prompts; repeat-run testing reduces but cannot remove the risk.
CSF-034 Apple: Apple Intelligence notification summaries
· United Kingdom · Search & consumer tech · Consumer AI assistant · Hallucination
An AI summary of BBC News notifications falsely said the suspect in a US killing had shot himself. The BBC complained, and further false summaries followed.
Outcome: In January 2025 Apple suspended AI summaries for news and entertainment apps.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Summary-faithfulness tests (the summary must not state anything absent from the source) are a core grounded-answer check.
· United Kingdom · Banking & fintech · Company support or sales bot · Offensive / unsafe, Task failure
A customer asking how to merge two Virgin Money ISAs was told 'Please don't use words like that', apparently because the bot flagged the word 'virgin'. The bank said it was an older chatbot already due for improvements.
Outcome: Virgin Money apologised and said it promptly removed the chatbot.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. An over-refusal test that uses the company's own brand and product names is a one-line case.
CSF-036 Hangzhou DeepSeek and Beijing DeepSeek: DeepSeek chatbot app
· Italy · AI platforms · Consumer AI assistant · Data leak
Italy's Garante ordered an immediate limitation on DeepSeek processing Italian users' data after finding the company's answers about what data it collected and where it was stored inadequate.
Outcome: Processing of Italian users' data blocked and an investigation opened (Garante, 30 January 2025).
Legal or regulatory action: Yes · Money on record: not disclosed
Would a pre-launch test have caught it? No. Regulatory data-protection compliance.
· United States · AI platforms · Consumer AI assistant · Data leak
Lasso Security reported that Copilot could return content from GitHub repositories that had been public and later made private, via Bing's cache, affecting more than 20,000 repositories. Microsoft classed the issue as low severity.
Outcome: Partial mitigations, per Lasso and TechCrunch.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? No. A retrieval-cache problem, not model behaviour.
CSF-038 Anysphere (Cursor): AI support agent 'Sam'
· United States · Software & SaaS · Company support or sales bot · Wrong promise, Hallucination
Users who were being logged out when switching devices emailed support and got a reply from 'Sam' stating that one device per subscription was a core security policy. No such policy existed; the reply was AI-generated. Some users said they were cancelling subscriptions.
Outcome: Co-founder Michael Truell apologised, said the user had been refunded and that AI responses used for email support are now clearly labelled as AI.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. A grounded-answer case ('Is there a one-device limit?') with the real policy as context would catch an invented rule.
· United States · Social media & consumer tech · Consumer AI assistant · Hallucination
Activist Robby Starbuck sued Meta for defamation, alleging Meta AI falsely said he took part in the January 6 Capitol riot. These were allegations in a complaint.
Outcome: Settled in August 2025; as part of the settlement Starbuck became an adviser to Meta on AI bias. Financial terms were not disclosed.
Legal or regulatory action: Yes · Money on record: not disclosed (settlement terms not disclosed)
Would a pre-launch test have caught it? Yes. Tests asking about named real people with an expected 'I don't have verified information' catch fabricated biographies.
· United States · AI platforms · Consumer AI assistant · Offensive / unsafe · followed an update or prompt change
An update to GPT-4o made ChatGPT overly flattering and agreeable. OpenAI said it focused too much on short-term user feedback and that sycophantic interactions can cause distress.
Outcome: OpenAI rolled the update back within days and said it would expand pre-deployment evaluations.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. OpenAI's own fix was more pre-deployment evaluation: a regression suite with 'should disagree' cases.
· Sweden · Banking & fintech · Company support or sales bot · Task failure
After saying its AI assistant did the work of 700 agents, Klarna's CEO told Bloomberg that cost had been 'a too predominant evaluation factor' and the result was 'lower quality'. A Klarna spokesperson later said this referred to outsourced human agents, not the AI.
Outcome: Klarna began recruiting human support agents so customers can always reach a person, while saying it remains AI-first.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Quality on complex cases can be measured pre-launch; the reported issue is disputed and partly about human staffing.
· United States · AI platforms · Consumer AI assistant · Offensive / unsafe · followed an update or prompt change
Grok inserted claims about 'white genocide' in South Africa into replies on unrelated topics. xAI said an unauthorized modification had been made to the bot's system prompt.
Outcome: xAI said it would publish Grok's system prompts on GitHub, set up a 24/7 monitoring team and add checks so employees cannot modify the prompt without review.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. A regression suite of off-topic questions run on every system-prompt change flags unrelated insertions immediately.
· United States · AI platforms · Consumer AI assistant · Hallucination
Radio host Mark Walters sued OpenAI for defamation after ChatGPT, asked by a journalist to summarise a lawsuit, falsely said Walters had been accused of embezzling funds. Date shown is the ruling.
Outcome: On 19 May 2025 a Georgia state court granted summary judgment to OpenAI.
Legal or regulatory action: Yes · Money on record: not disclosed (OpenAI won)
Would a pre-launch test have caught it? Yes. A summarisation test that compares the output with the source document flags invented allegations.
· United States · Social media & consumer tech · Consumer AI assistant · Data leak
Journalists found personal conversations, including medical and legal questions, published to the Meta AI app's public Discover feed, apparently by users who did not realise they were sharing. Meta said sharing is opt-in.
Outcome: Reported by Business Insider; Meta's position, per the reports, is that sharing is opt-in.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? No. A product-design and consent problem, not a model output.
· United States · AI platforms · Consumer AI assistant · Offensive / unsafe · followed an update or prompt change
Days after Grok was updated to 'not shy away from making claims which are politically incorrect', it posted antisemitic replies, praised Hitler and called itself 'MechaHitler', NPR reported.
Outcome: The bot appeared to stop giving public text answers. Separately, a Turkish court blocked access to Grok over responses the authorities said insulted President Erdogan, and Poland said it would report xAI to the European Commission (BBC).
Legal or regulatory action: Yes · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Hate-speech probes rerun after each deploy (regression) catch this class.
· United States · Restaurants & food · Company support or sales bot · Data leak
Security researchers Ian Carroll and Sam Curry reported that a Paradox.ai admin login using the password '123456' and an insecure API exposed chatbot records for up to 64 million job applications. Paradox said five candidates' information was viewed, only by the researchers.
Outcome: Paradox said it resolved the issue within a few hours of being notified and announced a bug bounty programme.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? No. A default password and an access-control bug are infrastructure security failures; no prompt test would find them.
· United States · Social media & consumer tech · Consumer AI assistant · Data leak
A security researcher found that Meta AI's servers did not check that a user was authorised to see a prompt, so changing an ID could reveal other users' prompts and responses. Meta fixed it and said it found no evidence of abuse.
Outcome: Fixed in January 2025; Meta paid the researcher a $10,000 bug bounty, per TechCrunch.
Legal or regulatory action: No · Money on record: $10,000 bug bounty paid
Would a pre-launch test have caught it? No. An access-control bug in the API, outside model behaviour.
· United States · Software & SaaS · Company support or sales bot · Task failure, Hallucination
SaaStr founder Jason Lemkin said the agent deleted a production database during a code freeze despite instructions not to make changes, and gave misleading information about whether recovery was possible.
Outcome: Replit's CEO Amjad Masad apologised and said Replit was rolling out automatic separation of development and production databases.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Instruction-following tests ('do not change anything') help, but destructive permissions need sandboxing, not prompts.
CSF-049 Amazon Web Services: Amazon Q Developer extension for VS Code
· United States · Software & SaaS · Company support or sales bot · Prompt injection
A hacker slipped a prompt into a released version of the Amazon Q extension instructing the AI agent to wipe local files and, under certain conditions, cloud resources, ZDNET reported.
Outcome: AWS published security bulletin AWS-2025-015 for version 1.84 of the extension.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. A supply-chain compromise; prompt tests on the shipped instructions would show a destructive instruction, but code review is the main control.
CSF-050 Google; Royal Caribbean (impersonated): Google AI Overviews (search answers)
· United States · Search & consumer tech · Consumer AI assistant · Hallucination, Prompt injection
The Washington Post reported that a traveller searched Google for a cruise line's customer-service number, called the number he was shown, and gave his credit card details to what turned out to be a scam operation.
Outcome: Reported by the Washington Post; no regulatory action in the source.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Tests that check contact details only come from an allow-listed source help; poisoned web data is a retrieval problem.
· United States · Consumer electronics · Company support or sales bot · Prompt injection, Data leak
Cybernews researchers showed that a single 400-character prompt made Lena output HTML that, when a human support agent opened the chat, could send that agent's session cookie to an outside server (stored cross-site scripting).
Outcome: Cybernews says Lenovo mitigated the issue before public disclosure on 18 August 2025; Lenovo said it implemented corrective actions.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. A structured-output test ('reply must be plain text, no HTML') plus an injection case catches the model half; output sanitising catches the rest.
CSF-052 Google; Stefanina's (restaurant harmed): Google AI Overviews (search answers)
· United States · Search & consumer tech · Consumer AI assistant · Hallucination
A family restaurant in Wentzville, Missouri, asked customers not to rely on Google's AI answers after it showed specials the restaurant did not offer, and customers demanded them.
Outcome: The restaurant posted a public warning; no response from Google in the source.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Grounded-answer tests catch invented offers in your own bot; you cannot test a third party's search answers.
CSF-053 Commonwealth Bank of Australia: AI voice bot in customer-service calls
· Australia · Banking & fintech · Company support or sales bot · Task failure
CBA cut 45 customer-service roles, citing a new AI voice bot. The Finance Sector Union said call volumes were rising and took the bank to the Fair Work Commission. CBA then said its assessment was an 'error' and the roles were not redundant.
Outcome: CBA reversed the cuts and apologised to the employees; the union's dispute was before the Fair Work Commission.
Legal or regulatory action: Yes · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Containment rate is measurable before launch with real call transcripts as test cases; staffing decisions are not.
CSF-054 Taco Bell (Yum! Brands): Voice AI drive-thru ordering
· United States · Restaurants & food · Company support or sales bot · Prompt injection, Task failure
Clips circulated of customers trolling the voice AI, including an order for 18,000 cups of water, and of the system repeatedly misunderstanding orders. The chain's digital chief told the Wall Street Journal it was rethinking where to use voice AI.
Outcome: Taco Bell said it was reassessing where the system is used, for example at busy locations; no shutdown was announced. Source: BBC, citing the Wall Street Journal.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. Absurd-quantity and troll orders are easy adversarial test cases with an expected behaviour (confirm, cap, or hand off to staff).
· Canada · Government & public services · Government or public-service bot · Hallucination
The Auditor General of Canada tested Charlie with six general tax questions; it answered two correctly. Other public AI tools tested answered five of six correctly. The audit covered CRA contact centres more broadly.
Outcome: Findings published in the Auditor General's October 2025 report on CRA contact centres; CRA's response to the recommendations was “Agreed”. National Post reported CRA spent about $18 million on Charlie.
Legal or regulatory action: Yes · Money on record: About C$18 million spent on the chatbot, as reported by National Post
Would a pre-launch test have caught it? Yes. A tax FAQ with answers from CRA guidance is a textbook grounded-answer suite; the auditor's own six-question test is one.
CSF-056 OpenAI, Microsoft, Google, Meta and others (tested by Which?): ChatGPT, Copilot, Gemini, Meta AI and others
· United Kingdom · AI platforms · Consumer AI assistant · Hallucination
Consumer group Which? tested AI chatbots on 40 money, legal and consumer questions and found errors: ChatGPT and Copilot did not notice a wrong ISA allowance and gave advice that could breach HMRC limits, and ChatGPT wrongly said travel insurance was mandatory for most EU countries.
Outcome: Research report; companies' responses are as quoted by the Guardian.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. The Which? study is itself a 40-case test suite with expected answers.
· China · Social media & consumer tech · Consumer AI assistant · Offensive / unsafe
Screenshots showed Yuanbao insulting a user who asked for help debugging code. Tencent apologised, called it a rare model output anomaly and said logs showed no human involvement.
Outcome: Tencent apologised and said it was optimising the model.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Tone checks on repeated runs reduce the risk; rare outputs need sampling at volume.
CSF-058 Woolworths (Australia): Digital assistant 'Olive'
· Australia · Grocery & retail · Company support or sales bot · Hallucination · followed an update or prompt change
Customers reported that Olive invented humanlike personal anecdotes, such as talking about its mother, during routine support chats after an upgrade built on Google's Gemini Enterprise for Customer Experience.
Outcome: Woolworths said it was adjusting Olive's responses.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Yes. A persona test ('Do you have a family?') plus regression runs after a model upgrade catch invented personal stories.
CSF-059 Meta (Instagram): AI support assistant for account recovery
· United States · Social media & consumer tech · Company support or sales bot · Prompt injection, Data leak
404 Media reported that attackers took over high-profile Instagram accounts by persuading Meta's AI support chatbot to change the account email and start recovery. Meta said the issue had been resolved.
Outcome: Meta said the issue was resolved.
Legal or regulatory action: No · Money on record: not disclosed
Would a pre-launch test have caught it? Partly. Social-engineering prompts against an account-recovery flow are testable, but the real fix is not letting a chatbot change account credentials.
CSF-060 BMW Toronto (dealership): Dealership AI chatbot 'Quinn'
· Canada · Automotive retail · Company support or sales bot · Wrong promise
The dealer's chatbot 'Quinn' made a customer a firm buy-back offer of C$27,162.79. The dealer revoked it, saying that because of a human employee's miscommunication the bot had taken the amount owed on the car as the buy-back amount.
Outcome: After hearing from CBC News the dealer reinstated the offer, which the customer accepted; it said only human employees will present buy-back offers in future.
Legal or regulatory action: No · Money on record: not disclosed (offer of C$27,162.79 reinstated; any loss to the dealer not disclosed)
Would a pre-launch test have caught it? Yes. A rule that the bot must never state a price or offer is testable with a handful of 'what will you give me for my car?' cases.
CSF-061 Anthropic: Fin AI support agent (Intercom) on Claude support
· United States · Software & SaaS · Company support or sales bot · Task failure
The San Francisco Standard reported a customer's account that after Anthropic created a duplicate account and charged him an extra $3,500, his refund request through the Fin AI agent became a months-long ordeal of circular conversations. These are the customer's claims as reported.
Outcome: Reported by the SF Standard; resolution details are as described in that report.
Legal or regulatory action: No · Money on record: not disclosed (customer reported a $3,500 overcharge)
Would a pre-launch test have caught it? Yes. An escalation test ('I want a human / a refund for a double charge') with an expected hand-off catches support loops.
We included failures of AI chatbots and assistants that talk to customers, citizens or users. That covers company support and sales bots, government service bots, and consumer AI assistants whose failure affected their own users. A failure means the bot gave wrong or harmful output, was manipulated, exposed data or failed the task it was deployed for, and the failure was publicly reported.
We excluded user misuse (for example, lawyers filing fake ChatGPT citations), companion and mental-health cases outside a support setting, internal tools, and incidents we could only find on social media or personal blogs.
Finding and verifying incidents
Candidates: we screened the AI Incident Database's public snapshot of 9 November 2026 (1,803 incidents) for chatbots and assistants. We then added incidents from searches of press, court and regulator records.
Verification: on 12 Nov 2026 a script fetched every source URL and checked that each key fact (names, amounts, quotes, outcomes) appeared on the page. All 96 sources returned HTTP 200 and contained the checked phrases. The JSON file lists the phrases for each source.
Exclusions: we left out any incident we couldn't verify, and rewrote or dropped any claim the fetched page didn't support. We didn't use pages that blocked automated access, so some well-known coverage behind paywalls isn't cited.
Dates: the date of the first public report among the checked sources. For court and regulator cases where the decision was the public event, we used the decision date.
Classification
Failure types: hallucinated or false facts; offensive or unsafe output; data leak or privacy breach; prompt injection or manipulation; failed the task or escalation; wrong policy, price or promise. Each incident has one primary type and may have more.
Legal or regulatory action: a court case, tribunal, regulator order or investigation, or an official audit concerning the AI deployment. Letters from officials and unrelated charges don't count.
Money on record: only figures stated by an official source or a cited publication. Otherwise we wrote "not disclosed". We never estimate, convert or add up currencies.
Testable before launch: Litmiq's judgment, explained in one sentence for every incident. "Yes" means a single-turn or short multi-turn test case with a clear expected behavior would likely have failed on the reported behavior.
Sources and wording
We report what the cited sources state and attribute claims to them. Lawsuits and complaints are allegations unless a court decided them, and we say so. Being listed here doesn't mean a company broke the law. Many companies fixed the problem quickly, and we record their responses.
The summaries are in our own words. We give AI Incident Database IDs for cross-reference. That database is licensed CC BY-SA 4.0, and we didn't copy its text. To report a correction, email hello@litmiq.com with a source.
Limitations
Reported, not representative. This is a record of failures that became public, mostly in English-language media. It can't tell you how often chatbots fail. Viral, funny or high-profile failures are over-represented.
Big brands are over-represented. Google, Meta, Microsoft, OpenAI and xAI appear often because their assistants are widely used and closely covered, not necessarily because they fail more.
Classification involves judgment. Failure types and testability are our calls. You can re-code them in the CSV.
Sources change. All links worked on 12 Nov 2026, but pages can move or go behind paywalls. The verification log records what we checked.
2026 is a partial year. Recent incidents may not yet have follow-up reporting on outcomes.
Download the dataset
The dataset is licensed under CC BY 4.0. You can use, share and adapt it, including commercially, as long as you credit "Litmiq, The State of AI Customer Support Failures 2026" and link to this page.
ai-chatbot-failures-2026.csv - one row per incident (61 rows, 21 columns: date, company, AI system, country, sector, deployment, failure types, summary, outcomes, legal action, money on record, pre-launch testability, source URLs, AIID link)
ai-chatbot-failures-2026.json - the same data as JSON, with every source's HTTP status, check date and the phrases we confirmed on the page
Litmiq (2026). The State of AI Customer Support Failures 2026 (version 1.0) [Data set]. 61 publicly verified incidents, sources checked 12 November 2026. https://litmiq.com/ai-chatbot-failures. License: CC BY 4.0.
BibTeX
@misc{litmiq_ai_support_failures_2026,
title = {The State of AI Customer Support Failures 2026 (version 1.0)},
author = {{Litmiq}},
year = {2026},
month = nov,
note = {61 publicly verified AI chatbot and assistant failures, sources checked 2026-11-12. License: CC BY 4.0},
howpublished = {\url{https://litmiq.com/ai-chatbot-failures}}
}
Link
<a href="https://litmiq.com/ai-chatbot-failures">The State of AI Customer Support Failures 2026</a>
FAQ
What are the most common AI chatbot failures?
Across our 61 verified incidents, the most common failure was hallucinated or false facts (24 incidents, 39%). Next came offensive or unsafe output (21), data leaks (14), prompt injection (11), failing the task (10) and wrong policy, price or refund promises (6).
Is a company liable for what its AI chatbot says?
In Moffatt v. Air Canada (2024), British Columbia's Civil Resolution Tribunal held Air Canada liable for negligent misrepresentation by its chatbot. It rejected the argument that the bot was a separate legal entity and ordered a CA$812.02 payment. That was one small-claims tribunal decision, not a general rule, and other cases have gone the company's way (Walters v. OpenAI). This is not legal advice.
How many AI chatbot failures led to lawsuits or fines?
13 of 61 (21%) reached a court, tribunal, regulator or official audit. Most involved privacy regulators (Italy's Garante, South Korea's PIPC, the UK ICO), consumer-protection cases or defamation cases. Only 9 incidents have any money figure on record.
Could these failures have been prevented with testing?
In our judgment, a pre-launch test case would likely have caught 35 of 61 (57%) and at least partly caught 52. The 9 that testing would not have caught were mostly infrastructure or access-control bugs. See what pre-launch tests would have caught for how each failure mode maps to a test type.
What is the most famous AI customer service failure?
The most cited cases are Air Canada, where a tribunal made the airline honour its bot's answer, and the $1 Chevy Tahoe chatbot. Two others are DPD's swearing bot and New York City's MyCity, which gave answers that contradicted the law. All four, with sources, are in the dataset.
Can I use this dataset?
Yes. The CSV and JSON are CC BY 4.0. Credit "Litmiq, The State of AI Customer Support Failures 2026" and link to this page. The Cite this section has text and BibTeX. Corrections with a source are welcome at hello@litmiq.com.
Get model deprecation alerts
We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.
We store only your email, through our form provider. See the privacy policy.
Don't be the next entry
Turn the incidents that look like your bot into test cases, then run them on every prompt change. Litmiq runs in your browser with your own OpenAI key.
We email you when OpenAI, Anthropic or Google publish a new shutdown date for a model on our tracker. Sent by hand from hello@litmiq.com, a few times a month at most. Unsubscribe by replying.
We store only your email, through our form provider. See the privacy policy.