Feature Flags and Gradual Rollouts for AI
Ship AI changes behind flags, roll out by percentage or cohort, shadow-test new prompts and models, and roll back in seconds.
You've rewritten Deskmate's ticket-summary prompt. It scores better on your golden dataset, the team likes the samples, and the diff is twelve lines. Shipping it the usual way means every customer gets the new behavior at the same moment, on inputs your eval set never covered. If it misbehaves on one large customer's oddly formatted tickets, you find out from their support lead, and the fix is another deploy.
AI changes deserve a gentler path to production. In this lesson you'll put prompts, models, and effort settings behind flags that carry configuration, roll changes out by cohort and percentage, run candidates silently in shadow mode on real traffic, and make rollback a config change that takes effect in seconds instead of a deploy.
Why AI changes need flags more than most code
A typical code change touches a known path: if you fix the date formatter, dates change and nothing else does. An AI change is different in three ways.
- It changes behavior on every input at once. A prompt edit or a model swap affects all requests the feature handles, including kinds you've never seen.
- Your evals cover a sample. A good golden dataset catches regressions on the cases you know about. Production has cases you don't.
- Many AI changes aren't code changes. Switching from one model to another, raising effort, or editing a prompt stored in a database can alter behavior without a deploy, a review, or a rollback button.
So the rule for AI features is simple: every change that can alter model behavior goes behind a flag. That covers more than new features:
| Change | Why it needs a flag |
|---|---|
| New AI feature | Classic launch control: staff first, then beta customers |
| New prompt version | Different outputs on every input |
| Model swap or upgrade | Different strengths, failure modes, latency, and cost |
| Effort or thinking change | Changes latency, cost, and sometimes quality |
| New context source (a new retrieval index, more history) | Changes what the model sees, so it changes what it says |
| Provider or routing change | New error modes and rate limits |
A flag here is a runtime switch your code consults on each request, stored somewhere you can change without deploying. A gradual rollout uses that switch to expose a change to a growing share of users while you watch the numbers.