Asynchronous Verified Semantic Caching for Tiered LLM Architectures

Researchers introduce Krites, an asynchronous caching system for Large Language Models that uses LLM judges to verify cached responses, improving efficiency without changing serving decisions. The system increases the fraction of requests served with curated static answers by up to 3.9 times while maintaining unchanged critical path latency.

• →Krites addresses the hard tradeoff between conservative and aggressive caching thresholds in LLM deployments.
• →The system uses asynchronous LLM judges to verify whether cached responses are acceptable for new prompts.
• →Approved matches are promoted to dynamic cache, expanding static reach over time without affecting real-time performance.
• →Testing shows up to 3.9x improvement in serving curated static answers for conversational and search workloads.
• →The approach maintains unchanged critical path latency while significantly improving cache hit rates.