Evaluating Extraction, Classification, and Generation
The right metrics for each feature type: field-level accuracy, precision and recall, confusion matrices, and pairwise comparison for open-ended text.
"Our triage feature is 91% accurate." It sounds like a result. It isn't, yet. If 85% of tickets are billing questions, a model that labels everything "billing" scores 85% without ever spotting a bug report. If your extraction feature gets 91% of fields right but invents a refund amount on the other 9%, it's a liability. And a drafted reply can't be "accurate" at all, because there's no single right answer to compare against.
Each feature type needs its own way of scoring. In this lesson you'll score Deskmate's three workhorse features the way each deserves: field-level outcomes for extraction, precision, recall, and a confusion matrix for classification, and pairwise comparison for generated replies. You'll also learn to report the results so they point at a fix, not just a number.
Match the metric to the shape of the output
The previous lessons gave you the building blocks: code-based checks for things you can verify deterministically and model-graded evals for judgment calls. This lesson is about what you aggregate those checks into. The right aggregate depends on what the feature produces:
| Feature type | Deskmate example | Unit you score | Metric that tells the truth | What a single accuracy number hides |
|---|---|---|---|---|
| Extraction | Pull order number, refund amount, purchase date from a ticket | Each field of each record | Per-field accuracy, split into missed vs. invented | Which field is broken, and whether errors are harmless blanks or confident fabrications |
| Classification | Route a ticket to billing, bug, how-to, account, or other | Each label decision | Precision and recall per class, confusion matrix | Rare classes failing completely behind a strong majority class |
| Generation | Draft a reply to the customer | A whole output, judged against criteria | Rubric pass rates, pairwise win rate against the current version | Everything; there's no reference to compare against |
The common thread: score at the level where you'd make a fix. You fix extraction one field at a time, so score fields. You fix classification by clarifying the boundary between two labels, so look at which pairs get confused. You fix generation by changing instructions and asking "is this better than before?", so measure exactly that.