When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression

Researchers present a diagnostic framework for evaluating KV cache eviction selectors in large language models, identifying three failure modes and demonstrating that value-aware ranking combined with evidence recovery achieves 72.6% accuracy on positive-margin test cases. The work addresses a critical bottleneck in long-context LLM inference by revealing why compression strategies succeed or fail.

This research tackles a fundamental efficiency challenge in modern LLM deployment. As language models handle increasingly longer contexts, the key-value (KV) cache consumed during decoding becomes a major computational bottleneck, limiting inference speed and consuming substantial memory bandwidth. While KV compression techniques attempt to reduce this overhead by selectively keeping only the most important tokens, previous work lacked diagnostic tools to explain why certain compression strategies work better than others.

The authors introduce a novel methodology that decouples diagnosis from optimization, systematically isolating three distinct failure modes: missing critical evidence, scoring irrelevant tokens highly, and inadvertently removing related information during projection. By holding selector configurations fixed while varying individual decision slots, they create a controlled experimental framework that reveals which components of a compression strategy drive success or failure. Their testing across LongBench, NeedleBench, and RULER benchmarks demonstrates that value-aware ranking—combining attention mass with estimated output impact—shows positive effectiveness on nearly 73% of high-margin cases, though only 32% of boundary cases, highlighting the challenge of near-threshold decisions.

For the AI infrastructure industry, this diagnostic approach has tangible implications. As inference costs remain a significant pain point for LLM deployment, understanding exactly why compression succeeds enables more targeted optimization rather than trial-and-error tuning. The finding that evidence recovery should precede output-value ranking provides a clear optimization hierarchy for practitioners building production systems. This work bridges academic analysis and practical deployment, offering engineers concrete guidance on cache compression priorities and helping identify which architectural choices genuinely improve inference efficiency versus which merely appear effective.

• →KV cache compression fails through three distinct mechanisms: missing critical evidence, misscoring tokens, and breaking evidence relationships during projection
• →Value-aware ranking combining attention mass and output impact shows 72.6% effectiveness on high-confidence cases but only 32.4% on boundary cases
• →Optimal cache compression strategy prioritizes evidence recovery, then output-value ranking, then preserving coupled evidence during projection
• →The diagnostic framework enables controlled evaluation by holding selector setup fixed while varying individual cache slots
• →Testing across LongBench, NeedleBench, and RULER demonstrates framework applicability across multiple benchmarks and model sizes