Failure pattern
Weak guardrails
A user could talk the model out of the rules it was deployed with.
Preventive controls from the record
| Incident | Control |
|---|---|
| INC-2024-0001 DPD | Run a fixed set of guardrail tests before every release of a customer-facing model, prompt or filter. |
| INC-2023-0002 Chevrolet of Watsonville (chatbot supplied by Fullpath) | Restrict a customer-facing bot to its task, and never let its text stand as an offer or commitment without a separate system of record. |
Controls are editorial summaries by Incidents.si, not statements by the organizations involved.
2 incidents with this pattern
- DPD customer service chatbot swears and criticises the company
A customer got DPD’s support chatbot to swear and write poems criticising the company. DPD said an error occurred after a system update and disabled the AI component.
DPD · AI component of the chatbot disabled; no financial or data harm reported
- Car dealership chatbot is talked into "agreeing" to sell an SUV for $1
Users instructed a dealership’s ChatGPT-powered chatbot to agree with anything, then had it accept a $1 offer for a new Chevrolet Tahoe. No sale took place.
Chevrolet of Watsonville (chatbot supplied by Fullpath) · No sale took place; dealerships began disabling the chatbot
No incidents match these filters.