Exact and Semantic Response Caching
Skip the model entirely for repeated requests: exact-match caches, semantic caches with embeddings, and the correctness traps of each.
Prompt caching makes a repeated prompt cheaper. Response caching makes it free: if you've already answered this exact request, return the stored answer and never call the model at all. No input tokens, no output tokens, and a response in milliseconds instead of seconds.
It sounds like a pure win, and for the right features it nearly is. For the wrong ones it's a correctness bug waiting for traffic: one customer sees an answer generated for another, a refund policy answer outlives the policy, or "how do I cancel my order?" gets the cached answer to "how do I cancel my subscription?". In this lesson you'll build an exact-match cache and a semantic cache for Deskmate, learn how to decide which features can safely use each, and set up the invalidation and measurement that keep a cache honest.
When skipping the model is safe
A response cache stores a model's output under a key derived from the request, and serves that output when a matching request arrives. It's the same idea as caching any expensive function, with one twist: model outputs aren't deterministic, so the cache doesn't return "the" answer. It returns an acceptable answer you generated earlier. That's only fine if any acceptable answer would do.
Ask three questions about a feature before caching it:
- Would the same input always deserve the same output? Classifying a ticket's text: yes. Drafting a reply that depends on the customer's account state: no, because the account changes.
- Is the output free of anything user-specific that isn't in the key? If the prompt includes the user's name, permissions, plan, or data, those must be part of the key, or one user's answer can leak to another.
- Can the answer go stale, and how fast? A summary of a ticket at a specific version never goes stale. An answer about your refund policy goes stale the moment the policy changes.
Here's how that plays out across typical features:
| Feature | Cache? | Notes |
|---|---|---|
| Classifying or tagging identical text | Yes, exact match | Duplicate tickets, re-imports, and retries hit constantly |
| Summarizing a specific document version | Yes, exact match | Key on a content hash, so edits miss naturally |
| Translating fixed UI strings or help articles | Yes, exact match | Same source text, same target language |
| Public help-center Q&A over a knowledge base | Maybe, semantic | Real savings, real risk; needs the guards below |
| Personalized replies, chat with account context | Rarely | Key would include so much state it never hits |
| Creative generation ("give me five headline ideas") | No | Users expect variety; a regenerate button that returns the same thing is broken |
Whatever you cache, a user-facing regenerate action must bypass the cache. If someone asks for another try, a cache that hands back the same output is a bug.