IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference

Researchers introduce IntentKV, a learned KV cache pruning technique that optimizes memory usage for multi-turn LLM agents without modifying the base model. The method achieves 23-30% reductions in peak request tokens and up to 92.6% fewer KV reads under tight memory budgets, addressing a critical bottleneck in long-horizon agent inference.

IntentKV tackles a fundamental constraint in scaling long-horizon AI agents: KV cache memory consumption, which has emerged as the primary performance bottleneck rather than parameter compute in extended reasoning tasks. Multi-turn agent workflows—involving tool calls, search results, and intermediate reasoning steps—create trajectories where both memory footprint and read bandwidth explode exponentially. This research demonstrates that intelligent pruning of key-value pairs significantly mitigates this bottleneck while preserving model accuracy.

The innovation lies in its architectural composability and practical efficiency gains. By maintaining a session-level QueryMemory that tracks cross-turn intent and using a memory-attention scoring mechanism, IntentKV identifies which historical tokens remain relevant to current queries. The technique employs slot-map redirection for eviction, allowing integration with existing prefix cache systems without architectural redesign. This compatibility matters significantly for production deployments where infrastructure investments already exist.

The experimental results underscore the magnitude of improvement possible: on the 100 longest test queries, worst-case peak request tokens dropped from 92.3k to 20.5k (77.8% reduction), while raw KV reads fell from 411M to 31M operations (92.6% reduction). These gains directly translate to lower latency, reduced memory requirements, and improved throughput in agent serving systems. For organizations deploying multi-turn agents at scale, this represents meaningful cost reduction and performance improvement.

Future developments likely involve applying similar intent-aware pruning to different model architectures and exploring whether this technique generalizes across diverse agent types and reasoning patterns. The compatibility with frozen base models enables rapid adoption without requiring model retraining.

• →IntentKV achieves 23-30% reduction in peak request tokens through learned pruning while maintaining baseline accuracy.
• →The method reduces worst-case KV reads by 92.6% on long-horizon tasks by intelligently identifying and removing irrelevant historical tokens.
• →Slot-map redirection design maintains compatibility with existing prefix cache systems, enabling practical deployment without infrastructure overhaul.
• →KV cache memory and bandwidth, not parameter compute, now represent the dominant bottleneck for long-horizon LLM agent inference.
• →The technique works with frozen base models, eliminating the need for expensive retraining across different model scales.