Evaluating Retrieval and RAG Quality
Measure retrieval separately from generation: recall@k, MRR, faithfulness, and answer relevance — with a small labeled set you can build in an afternoon.
You've added hybrid search, a reranker, query rewriting, and citations. Is the system better than it was a week ago? Without measurement, the honest answer is "it seems better on the five questions I tried." That's how teams ship a new chunking strategy that fixes the demo question and quietly breaks forty others.
RAG is harder to evaluate than most AI features because two different systems can fail: retrieval can miss the right passage, or generation can mishandle a passage it was given. A single "was the answer good?" score blends them together and tells you nothing about what to fix. In this lesson you'll build a small labeled set in an afternoon, measure retrieval with recall@k and MRR, measure generation with faithfulness and answer relevance, and run both every time you change the pipeline.
Measure the two halves separately
Every wrong RAG answer has a cause in one of two places, and the fixes are completely different:
| What happened | Where it failed | What fixes it |
|---|---|---|
| The right passage wasn't in the candidates | Retrieval (recall) | Hybrid search, query rewriting, better chunking, a different embedding model |
| The right passage was found but ranked too low to be used | Retrieval (ranking) | Reranking, a larger context budget |
| The right passage was in the prompt, but the answer contradicts or ignores it | Generation (faithfulness) | Prompt changes, a stronger model, higher effort |
| The answer is true to the sources but doesn't address the question | Generation (relevance) | Prompt changes, query rewriting |
| The docs don't cover it, and the model answered anyway | Generation (abstention) | Prompt changes, a similarity or rerank floor |
If you only measure end-to-end answer quality, a drop after a change could come from any row. If you measure retrieval on its own, you can change the reranker and know within minutes whether it helped, without paying for generation or grading.
So the eval has two layers. Retrieval metrics compare the chunks you retrieved against labels saying which content is relevant. They're computed in code, cheap, and deterministic. Generation metrics judge the answer against the retrieved context and the question. They usually need a grader model, so they cost more and need calibration.