Error Analysis: Read Your Outputs First
The highest-leverage quality habit: review real outputs, write open notes, group them into failure categories, and fix the biggest bucket first.
When an AI feature disappoints, the instinct is to reach for a metric: add a "helpfulness" score, wire up a dashboard, try a bigger model. Most of the time that's premature. You can't measure a failure you haven't named, and you can't name it until you've seen it. Generic scores tell you something is off; they almost never tell you what to fix.
Error analysis is the fix: read a batch of real outputs, write down what went wrong in plain words, group those notes into failure categories, and count them. It needs no framework and no grader, just a sample of real data, a spreadsheet's worth of discipline, and an afternoon. In this lesson you'll run it on Deskmate's drafted replies, build two small scripts that make it fast, and learn to turn the counts into a ranked list of fixes.
Why reading beats measuring at first
Teams that skip error analysis tend to build evals for failures they imagined. They write a tone grader because tone seemed important, a hallucination check because hallucination is what everyone talks about, and a helpfulness score because it sounds comprehensive. Then they discover, months later, that the most common real failure was something none of those checks could see: the draft asks the customer for an order number that's already in the ticket.
Reading outputs gives you three things a metric can't:
- Failure modes specific to your feature. "Asks for information already provided" isn't on any generic rubric, but it may be the top reason agents discard drafts.
- A sense of proportion. Ten minutes of reading tells you whether failures are rare edge cases or the norm. That changes what you build next.
- The right criteria. If you wrote a quality spec (see What Quality Means for an AI Feature), some of its criteria will turn out to matter a lot, some never fail, and you'll find failures it didn't cover. Error analysis is how the spec gets corrected.
It also scales down gracefully. Fifty outputs is enough to learn something real, and you can do it on the first day a feature has users, before a single eval exists.