LLM-as-judge

LLM-as-judge, also called model-graded evaluation, gives a model the input, the output, and a rubric, and asks for a verdict. It's how you scale evaluation for criteria that depend on meaning: did the summary invent facts, does the reply answer the actual question, is the tone right for an upset customer. It runs offline in your eval suite and online on sampled production traffic.

Judges are most reliable when the question is narrow and concrete. Ask the judge to list every claim in the answer the source doesn't support, rather than to rate faithfulness from 1 to 10, then let code turn that list into a pass or fail. Use structured outputs for the verdict, give the judge the same context the feature had, and check independent criteria in separate calls.

A judge is a model, so it has biases and blind spots, such as favoring longer answers or whichever option appears first in a comparison. Before trusting one, calibrate it against human labels on a set of real cases and measure how often they agree. Recheck agreement whenever you change the judge's model or prompt, and keep its known error rate in mind when you report results.