Collecting Feedback That Improves Your Product
Thumbs, edits, and implicit signals — what to collect, how to store it with the trace, and how to turn it into eval cases and fixes.
Almost every AI feature ships with a thumbs-up and thumbs-down button. Almost none of them improve anything. The clicks land in an analytics table, someone builds a chart of the ratio, the ratio wobbles, and nobody can say which prompt change would move it, because a thumbs-down with no context, no link to what the model was given, and no record of what the user did next is close to useless.
Feedback becomes valuable when you can trace every signal back to the exact generation it's about, and when someone regularly turns it into test cases and fixes. In this lesson you'll decide which explicit and implicit signals to collect for Deskmate's drafted replies, store them alongside the generation that produced the output, measure how much users edit, and build the weekly loop that converts feedback into eval cases.
Why most feedback buttons produce nothing
Feedback systems fail in four common ways, and each one has a design fix:
| Failure | What it looks like | Fix |
|---|---|---|
| No context | A thumbs-down with no idea what was wrong | Ask one follow-up question with specific reason options |
| No link to the trace | You know the user disliked something, but not what input, prompt version, or model produced it | Store a generation ID with every output and attach feedback to it |
| Too sparse | Most users never click the buttons | Collect implicit signals from what users do with the output |
| No owner | Feedback piles up in a table nobody reads | A recurring review that ends in test cases and fixes |
Explicit ratings are the signal most teams start with and the least useful on their own. The rest of this lesson is about fixing the other three.