A similarity decision replaces an exact-key decision
An ordinary cache retrieves a value by an exact key. A semantic cache embeds the new request, searches earlier request vectors, and reuses a result when the best match passes an application-defined acceptance rule.
The pattern addresses repeated natural-language questions that differ in phrasing. It also creates a new failure mode: two nearby vectors can represent requests whose correct answers differ.
Password instructions can be reusable; account status cannot
A request for password-reset steps and a request for where to change a password may safely share a public help-center answer. A question about an account suspension cannot reuse another user’s response merely because the questions are semantically similar.
The cache key therefore needs more than an embedding. Tenant, permissions, product version, locale, source freshness, and response type may all belong in the eligibility check.
Semantic caching is not prompt caching or retrieval
Prompt caching reuses computation for an identical input prefix, while semantic caching may return an earlier application result for different text. Retrieval fetches evidence for a new answer; a semantic cache may bypass generation altogether.
The semantic caching guide develops the design and failure cases, while the prompt caching guide covers the distinct provider-side mechanism. The common misuse is choosing one global similarity threshold without evaluating false hits on the decisions the application actually makes.
Invalidation is harder than matching
A cached answer may cease to be correct even when an incoming question remains similar. Source documents change, policies expire, and personalized facts move independently of the request wording.
Entries therefore need provenance and an invalidation key tied to the data behind the response. Evaluation should count harmful false hits separately from harmless misses because reusing the wrong answer is usually worse than paying to generate a fresh one.
Continue with these glossary entries: