Semantic Cache · Cosine Similarity Threshold 0.88
Incoming Query
"Can I return an item?"
→
Nearest Cached Query
0.91
vs threshold 0.88
CACHE HIT
Return stored response · skip retrieval + LLM call
Cuts
40–70%
of LLM API calls on the right workloads