Model-Graded Evals You Can Trust
Write grader prompts with clear pass/fail criteria, calibrate them against human labels, and track agreement so your grader stays honest.
Some quality criteria can't be reduced to a regex. Did the reply handle both things the customer asked about? Is every claim about plan features backed by the help article it cites? A human can answer these in seconds, but not for 400 test cases on every prompt change. So you ask a model to grade.
Model-graded evals (often called LLM-as-judge) are powerful and easy to get wrong. A grader that hasn't been checked against human judgment produces numbers that look precise and mean little, and teams end up optimizing for the grader's quirks instead of their users. In this lesson you'll build graders for two of Deskmate's drafted-reply criteria, structure them so code makes the final call, measure how often they agree with a careful human, and use that measurement to correct the pass rates you report.
What to hand to a model grader
Give a grader only the criteria code can't check. If a criterion is about form or exact content (length, format, a required phrase, an invented order number), a code-based check is cheaper, faster, and more reliable. Graders are for criteria that depend on meaning:
- Coverage: does the output handle every request, step, or question in the input?
- Support: is each claim backed by the source material the feature was given?
- Policy in any wording: does it promise money or dates, however phrased?
- Fit for the audience: does the tone match a frustrated customer, or a terse one?
Three rules shape every grader in this lesson:
- One criterion per grader. A prompt that scores coverage, correctness, and tone at once produces verdicts where one quality bleeds into another, and a failure doesn't tell you what to fix. Separate graders, separate calls.
- Binary outcomes. Pass or fail, defined in writing. A 1–10 score is hard for a model to apply consistently and hard for you to act on.
- Give the grader everything a human would need. To judge support, it needs the help articles. To judge coverage, it needs the full ticket, not a summary of it.
Grading single outputs is the focus here. Comparing two versions of a feature side by side is covered in Evaluating Extraction, Classification, and Generation, and grading the multi-step runs of an agent is a different problem again, which the Agent Builder Course covers in evaluating agent trajectories.