Failure pattern

Weak guardrails

A user could talk the model out of the rules it was deployed with.

Preventive controls from the record

IncidentControl
INC-2024-0001
DPD
Run a fixed set of guardrail tests before every release of a customer-facing model, prompt or filter.
INC-2023-0002
Chevrolet of Watsonville (chatbot supplied by Fullpath)
Restrict a customer-facing bot to its task, and never let its text stand as an offer or commitment without a separate system of record.

Controls are editorial summaries by Incidents.si, not statements by the organizations involved.

2 incidents with this pattern

  1. INC-2024-0001
    DPD customer service chatbot swears and criticises the company

    A customer got DPD’s support chatbot to swear and write poems criticising the company. DPD said an error occurred after a system update and disabled the AI component.

    DPD · AI component of the chatbot disabled; no financial or data harm reported

    SEV4Model behaviorReported
  2. INC-2023-0002
    Car dealership chatbot is talked into "agreeing" to sell an SUV for $1

    Users instructed a dealership’s ChatGPT-powered chatbot to agree with anything, then had it accept a $1 offer for a new Chevrolet Tahoe. No sale took place.

    Chevrolet of Watsonville (chatbot supplied by Fullpath) · No sale took place; dealerships began disabling the chatbot

    SEV4Model behaviorReported