Experiments and A/B Tests for AI Changes
Test prompt and model changes on real users: choosing metrics that reflect value, sample sizes, guardrail metrics, and reading noisy results honestly.
Your new reply prompt wins 64% of pairwise comparisons offline, passes every gate in CI, and the team wants to ship it. But the offline judge measured what you think a good draft looks like. Whether support agents send more of those drafts, resolve tickets faster, or quietly rewrite them is a different question, and only real usage can answer it.
An A/B test (a controlled experiment) answers it by randomly splitting users between the current version (the control) and the new one (the treatment) and comparing outcomes. Random assignment is what lets you credit a difference to your change instead of to which customers happened to see it. This lesson covers experiments on AI changes: metrics that reflect value, how much traffic you need, guarding against wins that cost too much, reading noisy results honestly, and what to do when you lack the traffic for an A/B test.
When an experiment is worth running
Not every change needs one. Experiments take weeks, and while one runs, some users get the worse version. Decide based on what kind of change it is:
| Situation | What's enough |
|---|---|
| Fixing a clear failure (drafts promising refunds) | Offline evals and the CI gate. Don't make users wait weeks for a bug fix. |
| Cost or latency change with the same quality bar (moving triage to a smaller model) | Offline evals, then a monitored gradual rollout |
| A change meant to make the feature more useful (new reply style, adding retrieval, a different model for drafts) | An experiment, because "more useful" is about user behavior |
| A new AI feature replacing a manual workflow | An experiment, often with a long-running holdout (see below) |
Offline evals tell you a change is safe to try. Experiments tell you whether it helps. The quality spec from What Quality Means for an AI Feature already separates these: output criteria are leading indicators, and outcome metrics are what experiments measure.