From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings

Researchers propose semantic caching solutions for large language models to improve response times and reduce costs by reusing semantically similar requests. The study proves that optimal offline semantic caching is NP-hard and introduces polynomial-time heuristics and online policies combining recency, frequency, and locality factors.

• →Semantic caching for LLMs can significantly improve response times and reduce operational costs by reusing similar requests.
• →Implementing optimal offline semantic caching policies is proven to be NP-hard, requiring alternative approaches.
• →Researchers developed polynomial-time heuristics and online cache policies that combine recency, frequency, and locality.
• →Frequency-based policies serve as strong baselines, but novel variants show improved semantic accuracy.
• →The research identifies substantial room for future innovation in LLM caching systems.