Semantic caching

An exact-match cache only helps when identical text arrives twice. A semantic cache embeds each incoming question, looks up the most similar previously answered question in a vector index, and returns the stored answer if the two are close enough. Paraphrases such as "forgot my password" and "how do I reset my password" can then share one answer.

The payoff is skipping the whole pipeline, retrieval and generation, for questions people ask over and over, saving both cost and latency. It fits public, non-personalized questions with stable answers, like help-center FAQs.

The danger is that similar isn't the same. Embeddings capture topic well and decisive details poorly, so "cancel my order" and "cancel my subscription", a question and its negation, or two different order numbers can look almost identical. A wrong hit is a confident wrong answer with no model involved to notice. Scope every lookup by tenant so one customer never receives another's answer, include the knowledge-base version so answers expire when content changes, tune a conservative threshold on labeled question pairs, and refuse hits when numbers or negations differ. For high-stakes topics, a curated FAQ with human-approved answers is often the safer version.