RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction

RetentiveKV introduces an entropy-driven optimization method for multimodal large language models that achieves 5x KV cache compression and 1.5x decoding acceleration by reformulating token eviction as continuous memory evolution rather than discrete pruning. The approach addresses limitations of existing compression methods by accounting for visual tokens that gain importance later in decoding and preserving spatial continuity of visual information.

Multimodal large language models face significant computational bottlenecks when processing extended visual contexts, with KV caches consuming disproportionate memory and computational resources. RetentiveKV addresses this by departing from traditional discrete pruning approaches that assume token importance remains constant throughout inference. The research identifies a critical gap: visual tokens often exhibit deferred importance, initially appearing low-salience but becoming contextually critical during later decoding stages. This insight challenges the foundational assumptions underlying existing compression methods.

The technical innovation leverages state space models to transform KV eviction into a continuous process governed by information entropy. Rather than permanently removing low-attention tokens, RetentiveKV integrates them into a continuous state space where they remain dynamically reactivable when their semantic relevance emerges. This preserves the spatial continuity inherent in visual information, avoiding the fragmentation that discrete pruning introduces.

The performance metrics demonstrate substantial practical impact: 5x KV cache compression directly reduces memory requirements for inference, while 1.5x decoding acceleration improves throughput. These gains matter for deployment scenarios where computational resources are constrained—edge devices, real-time applications, and large-scale inference services. The methodology suggests a broader shift in how the AI community approaches resource optimization: from destructive truncation toward intelligent memory management that maintains information potential.

Future developments will likely explore how entropy-driven approaches generalize across different model architectures and whether similar techniques apply to language-only models or other modalities beyond vision and text.

• →RetentiveKV achieves 5x KV cache compression and 1.5x decoding speedup through entropy-guided state space optimization
• →The method preserves visual token importance by treating eviction as continuous memory evolution rather than discrete truncation
• →Addresses the deferred importance problem where visual tokens gain semantic relevance during later decoding stages
• →Maintains spatial continuity of visual information, avoiding fragmentation caused by traditional pruning approaches
• →Demonstrates significant practical efficiency gains relevant for edge deployment and real-time multimodal inference applications