Reranking

First-stage retrieval, whether vector, keyword, or hybrid, is built for speed across a large collection, so its ordering is approximate. A reranker takes the top few dozen candidates and scores each one against the query directly, usually with a cross-encoder that reads query and passage together, and then reorders them. You keep only the best few for the prompt.

Rerankers matter because the model can only use what you send it, and you can only send a handful of passages without bloating the prompt. Pushing the truly relevant passage from position 15 into the top five is often the biggest single quality gain in a RAG pipeline. Hosted reranking APIs and open-weight reranker models exist; an LLM can also act as a reranker, at higher cost and latency.

The trade-off is latency and money per query, so rerank a shortlist, not the whole corpus. Reranker scores are also more meaningful than fusion scores, which makes them useful for deciding that nothing relevant was found.

Retrieve generously in the first stage, typically a few dozen candidates, then rerank down to what fits. Measure whether recall at your final cutoff actually improves.