When AI Breaks: Incidents and Provider Outages
Plan for provider outages, quality regressions, and runaway costs: detection, degraded modes, incident runbooks, and honest communication.
At some point, your AI feature will break while you're asleep. The model provider will have a bad hour and return errors on every third request. Or nothing will error at all, and summaries will quietly start dropping the customer's second request after a change nobody connected to it. Or a single tenant's integration will loop, and your AI bill for the day will pass your bill for the month before anyone looks.
Normal operations practice covers the first case partly and the other two barely. In this lesson you'll learn the three shapes AI incidents take, set up detection for each, design degraded modes before you need them, write a runbook your team can follow at 3 a.m., and communicate with customers in a way that keeps their trust.
The three kinds of AI incident
Most incident tooling assumes failures are loud: exceptions, 500s, timeouts. AI features fail loudly only some of the time.
| Kind | What it looks like | Typical causes | Does normal monitoring see it? |
|---|---|---|---|
| Availability | Errors, timeouts, very slow responses | Provider outage or overload, rate limits, network problems, an expired API key | Yes, if you log failed calls |
| Quality | Calls succeed, outputs are worse | Your prompt or model change, a broken retrieval index, changed input formats, an injection campaign | No. Every request is a 200 |
| Cost | Calls succeed, spend climbs fast | A retry loop, a runaway integration, abuse, a change that broke prompt caching | No, until the invoice |
The second and third rows are why AI incidents need their own detection. A summary that invents a refund returns HTTP 200 in 900 milliseconds. A cache-busting change makes every request succeed at several times the normal price. If your alerting only looks at error rates, both look like a good day.