Risk Assessment for AI Features
A lightweight method to find what could go wrong with a feature — who's harmed, how badly, how likely — and pick proportionate mitigations.
The Deskmate team is a week from shipping drafted replies: the model reads a support ticket and writes a response for the agent to send. The demo is great. Then someone asks, "What's the worst thing this could send to a customer?" and the room goes quiet, because nobody has thought about it on purpose.
A risk assessment is the deliberate version of that question, and it doesn't need a committee. In this lesson you'll learn a method that takes under an hour per feature: describe the feature as a system, list how it can fail, score each failure by who gets hurt, how badly, and how often, then pick proportionate mitigations. You'll finish with a risk register that lives in your repo and fails CI when a serious risk has no defense.
Why AI features need their own risk pass
Normal engineering review asks "what if this code has a bug?" AI features can misbehave while the code is perfectly correct:
- Output varies. The same input can produce different output after a model update or a one-word prompt change. You're assessing a distribution of behaviors, not one code path.
- Errors look plausible. A broken function throws. A wrong model writes a confident, well-formatted paragraph.
- Every piece of text is an input channel. Ticket bodies, uploaded files, retrieved help articles, and web pages can all steer the model, not just the user typing into the box.
- Mistakes repeat at scale. A flaw that shows up in one request in a thousand isn't rare once the feature handles 50,000 requests a month.
- People over-trust it. Automation bias is the tendency to accept a machine's suggestion without checking it. The better a feature usually is, the less carefully people check it on the day it's wrong.
A grammar suggestion and an automated refund decision carry wildly different stakes. The point of an assessment is to find out which one you're building, and to spend safety effort where it matters instead of spreading it evenly.