Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation

Researchers introduce Moment-KV, a momentum-based compression technique that optimizes Key-Value cache usage during LLM decoding phases. The method improves long-generation task performance by 2.3-3.2% while maintaining latency by dynamically tracking token importance through temporal attention patterns rather than static heuristics.

Moment-KV addresses a critical infrastructure challenge in large language model deployment. KV cache consumption directly constrains how long an LLM can generate text before running out of memory, making it a fundamental bottleneck for production systems handling extended outputs. Unlike previous approaches that apply uniform compression across model stages, this work recognizes that prefill and decoding phases have distinct characteristics—prefill requires full context fidelity while decoding can tolerate selective compression.

The innovation lies in modeling token importance as a dynamically evolving metric rather than relying on rigid recency windows. By aggregating attention signals with exponential decay, Moment-KV captures tokens that maintain consistent influence across long horizons while pruning noise from transient bursts. This temporal modeling aligns with how transformer attention actually functions, where critical tokens receive sustained focus while local reasoning involves short-term patterns.

For the AI infrastructure sector, this advancement has immediate practical implications. Deployment costs correlate directly with memory requirements—reducing KV cache size enables either longer context windows on existing hardware or cost reduction through smaller GPUs. The 2.3-3.2% performance improvement without latency penalties suggests the method achieves meaningful compression rates while preserving generation quality, making it production-ready.

The research validates a broader trend: static optimization heuristics increasingly yield to learned or analytically-driven dynamic approaches. Future work likely extends this momentum-based framework to other cache layers and explores adaptive decay parameters. This positions memory-efficient generation as a differentiating capability for edge deployment and cost-conscious cloud providers.

• →Moment-KV improves long-generation LLM performance by 2.3-3.2% through momentum-driven KV cache compression during decoding phases
• →The method models token importance dynamically using temporal attention aggregation with decay, rather than static recency heuristics
• →Preserving prefill cache while compressing only decoding cache avoids performance degradation from corrupted context
• →Reduced KV cache size directly lowers memory costs and enables longer context windows on existing hardware
• →Momentum-based temporal modeling captures both long-term token influence and recent relevance patterns in attention dynamics