#kv-caching News & Analysis

5 articles tagged with #kv-caching. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

5 articles

AIBullisharXiv – CS AI · Jun 97/10

🧠

SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance

Researchers introduce SIFT, a novel optimization technique for Retrieval-Augmented Generation (RAG) systems that exploits attention patterns to accelerate LLM prefill computation. By storing only compact bit vectors of high-attention locations rather than full KV tensors, SIFT achieves 1.71x faster time-to-first-token while reducing storage by up to 24,000x and maintaining accuracy within 1% of standard methods.

AIBullisharXiv – CS AI · Jun 97/10

🧠

Enabling KV Caching of Shared Prefix for Diffusion Language Models

Researchers introduce bicache, a novel KV caching technique that enables efficient serving of diffusion language models (DLMs) with shared prefixes. Unlike traditional LLMs, DLMs use bidirectional attention, which invalidates conventional caching methods and causes accuracy collapse. Bicache dynamically identifies safe layer depths for prefix reuse, achieving 36-98% throughput improvements.

AIBullisharXiv – CS AI · Jun 27/10

🧠

WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering

Researchers introduce WaveFilter, a training-free framework that uses wavelet transforms to optimize Key-Value cache filtering in Diffusion Large Language Models, addressing computational bottlenecks in long-context processing. The technique enables sparse KV caching to maintain generation quality while reducing inference latency, offering plug-and-play compatibility with existing LLM architectures.

AIBearisharXiv – CS AI · Apr 207/10

🧠

The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference

Researchers have discovered that FP16 floating-point precision causes systematic numerical divergence between KV-cached and cache-free inference in transformer models, producing 100% token divergence across multiple architectures. This challenges the long-held assumption that KV caching is numerically equivalent to standard computation, with controlled FP32 experiments confirming FP16 non-associativity as the causal mechanism.

AIBullisharXiv – CS AI · Jun 236/10

🧠

Recency/Frequency Adaptive KV Caching for Large Language Model Serving

Researchers propose an adaptive key-value caching strategy for large language models that dynamically allocates cache space based on recency and frequency patterns, improving upon traditional LRU eviction policies. The approach demonstrates up to 10.8% improvement in cache hit rates and 12.6% reduction in time-to-first-token on synthetic workloads, with more modest gains on real-world conversation data.