Failure pattern

Ungrounded answers

The model stated policies or facts that had no basis in an authoritative source.

Preventive controls from the record

IncidentControl
INC-2025-0001
Cursor (Anysphere)
Label AI replies, limit policy statements to documented policies, and escalate account and billing questions to a person.
INC-2024-0003
City of New York
Before launch, test against a set of known legal questions reviewed by the responsible agencies, and answer from the source text or not at all.
INC-2024-0002
Air Canada
Ground policy answers in the authoritative policy text and test them against it; route what cannot be grounded to a person.

Controls are editorial summaries by Incidents.si, not statements by the organizations involved.

3 incidents with this pattern

  1. INC-2025-0001
    Cursor’s AI support bot invents a one-device policy that does not exist

    A support bot signing as "Sam" told a user that Cursor allows one device per subscription. No such policy existed, and a co-founder corrected it publicly.

    Cursor (Anysphere) · Users announced subscription cancellations after receiving a fabricated policy

    SEV4Model behaviorReported
  2. INC-2024-0003
    New York City’s business chatbot gives advice that would break the law

    An investigation found that New York City’s official chatbot for business owners gave answers that contradicted city and state law. The city kept it online with disclaimers until 2026.

    City of New York · Wrong legal guidance on tips, cash payments and housing vouchers given to business owners; chatbot taken down in February 2026

    SEV3Model behaviorReported
  3. INC-2024-0002
    Tribunal holds Air Canada liable for wrong fare advice from its chatbot

    A British Columbia tribunal found Air Canada liable for negligent misrepresentation after its website chatbot told a customer he could claim a bereavement fare retroactively.

    Air Canada · Airline ordered to compensate one customer for the fare difference

    SEV4Model behaviorConfirmed