Mean reciprocal rank (MRR)

For each query, find the position of the first relevant result and take its reciprocal: 1 if it's first, 0.5 if second, about 0.33 if third, and 0 if it doesn't appear within the cutoff. Average across all queries and you get mean reciprocal rank. Often it's reported with a cutoff, written MRR at k.

MRR captures something recall at k misses: order. Two systems can both have the right passage somewhere in the top ten, but the one that puts it first is better, especially when users scan a result list or when you only send the top few passages to the model.

It has limits. MRR only looks at the first relevant result, so it doesn't reward finding several relevant passages, which matters when an answer needs facts from multiple places. A single high-ranked hit can also hide a weak tail. Use it alongside recall at k rather than instead of it.

MRR is especially useful for judging rerankers and query rewriting, whose main job is moving the right result up the list. If recall barely changes but MRR rises, the change is doing its job.