Pairwise evaluation

Pairwise evaluation shows a judge, human or model, two responses to the same input and asks which better meets stated criteria. Across a test set, you get a win rate: how often the candidate beats the baseline, with ties counted separately.

Comparisons are easier to make consistently than absolute scores. Asked to rate a reply from 1 to 10, judges cluster in the middle and drift over time; asked which of two replies better answers the customer, they're more stable. And "is this better than what we ship today?" is usually the exact question you need answered when you change a prompt or model.

Position bias is the classic trap: judges tend to favor whichever option sits in a particular position. Judge each pair twice with the order swapped and count a win only when both orders agree, treating disagreement as a tie. Remember that pairwise results show direction, not acceptability. A candidate can beat a weak baseline most of the time and still fall below your quality bar, so run your absolute checks on it too. In production, the same idea can run as an occasional side-by-side choice shown to opted-in users.