KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving

Researchers present KV-RM, a runtime optimization that manages KV-cache memory movement in static-graph LLM decoders, achieving better throughput and reduced latency variability without sacrificing the predictability benefits of static graph execution. The approach decouples logical KV histories from physical storage through a block pager and merge-staged transport mechanism, demonstrating practical improvements on multi-GPU systems.

KV-RM addresses a fundamental tension in LLM serving infrastructure between predictability and efficiency. Static-graph decoders offer deterministic performance characteristics essential for production systems, but struggle with the inherent variability of online inference workloads where request lengths and completion times differ significantly. This paper demonstrates that much of this inefficiency stems not from kernel design limitations but from suboptimal memory management strategies.

The technical approach separates concerns elegantly: rather than forcing physical KV-cache layouts to match logical request histories, KV-RM maintains flexible physical storage while presenting a consistent interface to the decoder graph. The block pager and merge-staged transport mechanism efficiently coalesce fragmented memory accesses into larger transfer operations, reducing memory overhead and latency spikes without requiring architectural changes to existing kernels.

For production LLM inference systems, this represents a meaningful optimization lever. The improvements in tail latency under production-trace replay suggest practical benefits for serving platforms where consistent response times matter as much as throughput. The optional far-history summarization capability provides a pathway toward handling longer context windows without abandoning the static-graph execution model.

Developers and infrastructure providers should monitor whether this approach generalizes across different hardware platforms and LLM architectures. The core insight—that cache movement management can absorb variability better than static kernel shapes—may influence how future serving systems balance flexibility and predictability. Success here could shift industry practices away from over-provisioned memory reservations toward more efficient adaptive memory management within static execution frameworks.

• →KV-RM decouples logical KV histories from physical storage through block paging, enabling flexible memory management within static-graph constraints
• →Merge-staged transport coalesces fragmented KV mappings into large transfer groups, reducing memory overhead and latency outliers on production workloads
• →The approach improves mixed-length decoding throughput and tail latency without requiring changes to core attention kernels or static-graph execution models
• →Optional bounded far-history summaries can extend context handling capabilities using the same runtime interface
• →Results suggest cache movement optimization is a more effective boundary than kernel shape for recovering runtime flexibility in static LLM serving