Drift

Drift is what happens when the world your feature runs in stops matching the world you tested. Input drift means the traffic changes: a large new customer, a product launch, a new language, a seasonal surge in billing questions. Output drift means what the feature produces changes: the mix of labels shifts, replies get longer, refusals rise. It happens even with a fixed model, because your users and data keep moving.

Changes on the provider side cause shifts too. Providers release new model versions and retire old ones, and a forced migration can change behavior across a feature. Treat every model change as a release that goes through your evals, rather than a configuration tweak.

Drift isn't automatically bad, but it means your golden dataset describes reality less well than it did. Watch for it with simple signals: label distributions compared against a baseline window (often summarized with a statistic such as the population stability index), output length, refusal rate, validation failures, and feedback rates. When something moves, read a sample before reacting, then refresh your eval set with the new kind of traffic.