A/B testing (for AI features)

An A/B test randomly assigns users or accounts to the current version or a candidate, such as a new prompt, model, or retrieval setup, and compares outcome metrics between the groups. It answers what offline evals can't: does this change actually help real people doing real work?

For AI features, the metrics that matter are behavioral. Do agents send the drafted reply, edit it heavily, or throw it away? Do customers resolve their question without opening a ticket? Do people use the feature again next week? Pair those with guardrail metrics such as cost per request, latency, error rate, and complaints, so a win that doubles spend or slows the page doesn't ship unnoticed.

Several things make AI experiments tricky. Output varies even within one arm, so effects can be small and need enough traffic to detect. Randomize by user or account, not by request, so nobody sees both versions. Choose the primary metric and the test length before you start, instead of stopping when the numbers look good. And run your offline evals first: an A/B test should confirm a change that already meets your quality bar, not find out whether it's safe.