Query rewriting

What users type is often a poor search query. In a conversation, a follow-up like what about for annual plans depends on earlier turns. A question may be long and rambling, use different vocabulary from your docs, or bundle two questions in one. Query rewriting asks a model to produce better queries before you search.

Common forms include rewriting a follow-up into a standalone question using the conversation history, generating a few alternative phrasings and merging the results, splitting a compound question into separate searches, and extracting filters such as product, date range, or plan into structured fields. A small, fast model usually handles this well.

The costs are an extra model call before retrieval, which adds latency to time to first token, and the risk that a rewrite changes the meaning or drops a crucial detail, such as an exact error code. Keep the original query in the mix rather than replacing it, and log both versions so you can spot bad rewrites.

Rewriting helps most in conversational interfaces and with vague questions. Measure it with your retrieval eval; on clean, specific queries it may add latency without improving recall.