Monitoring Quality in Production
Sample live traffic for grading, watch for drift, alert on quality drops, and close the loop from production failures back into your test set.
Your eval suite passed, the rollout went smoothly, and Deskmate's drafted replies have been live for a month. Are they still good? You don't actually know. Since launch, your biggest customer started routing German tickets through Deskmate, the help center gained thirty new articles, and a pricing change made "can I downgrade?" the most common question of the week. None of that is in your golden dataset. Your feature isn't failing loudly. If it's failing, it's doing so quietly, in a slice you aren't watching.
This lesson builds the production half of your quality system: sampling live traffic and grading it with the same criteria you use offline, watching cheap signals on every request, detecting when inputs or outputs drift, alerting on real quality drops without paging anyone for noise, and feeding what you find back into your test set.
What offline evals can't see
Offline evals answer "is this version good on the cases we have?" Production adds four things your test set can't contain:
- New inputs. Users bring languages, topics, and phrasings that weren't around when you built the dataset.
- A changing world. Help articles get edited, prices change, and a reply that was correct in March is wrong in May without a single line of your code changing.
- Changes you didn't make. A dependency update, a retrieval index rebuild, or a provider-side change can shift behavior. That's why your eval-driven workflow includes a scheduled run on the current version.
- Scale. A failure that happens once in 2,000 requests will probably never appear in a 200-case test set. At 5,000 requests a day, it happens two or three times daily.
Monitoring doesn't replace offline evals. It's how you find out which cases your offline evals are missing.