Reranking and Query Rewriting

Your RAG pipeline works on the questions you tested it with. Then real users arrive. One asks "why is it doing that again??" in the middle of a conversation. Another asks a perfectly clear question, and the chunk that answers it is sitting at position 14 in your results, just outside the five you send to the model. Both failures happen before the model writes a word, and no prompt change will fix them.

This lesson covers the two upgrades that fix them. Reranking reorders a broad set of candidates with a slower, more accurate model so the best chunks land on top. Query rewriting turns vague, conversational, or compound questions into queries your search index can actually match. You'll build both for Deskmate's help center, see what each costs in latency and money, and learn when to skip them.

Why first-stage retrieval leaves good results behind#

Vector search is fast because of how embeddings work: the question and every chunk are each turned into a vector separately, ahead of time for chunks and once for the query, and then compared with a cheap similarity calculation. Models that encode each side independently like this are called bi-encoders. Independence is what makes searching millions of chunks in milliseconds possible, and it's also the weakness. The model squeezes each chunk into a single vector without knowing what question will be asked, so fine distinctions get blurred.

Ask Deskmate "can I get a refund if I cancel an annual plan halfway through?" and a bi-encoder sees a question about refunds and annual plans. It will happily rank "How to cancel your plan", "Annual vs. monthly billing", and "Refund policy for monthly plans" above the one paragraph that answers the actual question, because all of them are about the same topics.

A cross-encoder reads the question and a candidate chunk together and outputs a relevance score for that pair. Because it sees both at once, it can tell that a chunk about monthly refunds doesn't answer an annual-plan question. It's far more accurate and far too slow to run over your whole index, which is why the standard design has two stages:

1. Retrieve broadly with fast search (vector or hybrid): take the top 30 to 100 candidates. The goal here is recall, making sure the right chunk is somewhere in the list.
2. Rerank narrowly: score every candidate against the question with a cross-encoder or an LLM, and keep the top 5 to 10. The goal here is precision, making sure what you send to the model is the best of what you found.

This split also changes how you tune retrieval. Without a reranker, you're stuck choosing a small k (and missing answers) or a big one (and drowning the model). With a reranker, first-stage k can be generous because the reranker cleans up.