OjaKV: Context-Aware Online Low-Rank KV Cache Compression

OjaKV introduces a novel framework for compressing key-value caches in large language models through online low-rank projection, addressing a critical memory bottleneck in long-context inference. The method combines selective full-rank storage for important tokens with adaptive compression for intermediate tokens, maintaining accuracy while reducing memory consumption without requiring model fine-tuning.

The memory demands of long-context language models present a fundamental engineering challenge that directly impacts inference costs and accessibility. OjaKV addresses this by recognizing that KV-cache compression need not apply uniformly across all tokens—a strategic insight that separates this approach from prior static compression methods. By preserving first and recent tokens in full precision while compressing intermediate ones, the framework maintains critical attention anchors while reducing overall memory footprint.

The innovation leverages Oja's algorithm for online principal component analysis, enabling the compression basis to adapt as context evolves during inference. This contrasts with offline-learned subspaces that degrade under distribution shifts, a persistent limitation in production deployments handling diverse inputs. The comprehensive updates during prompt prefilling and lightweight periodic updates during decoding balance computational efficiency with subspace alignment.

For practitioners deploying large models, this work reduces operational friction significantly. A 32GB KV-cache requirement becoming orders of magnitude smaller directly translates to lower GPU memory requirements, enabling inference on more accessible hardware and reducing cloud infrastructure costs. Compatibility with FlashAttention ensures integration into existing optimization pipelines without architectural changes.

The benchmarks showing particular gains on complex reasoning tasks with very long contexts suggest the method captures something fundamental about attention importance. Future research should examine how this scales across different model architectures and whether the online adaptation overhead varies predictably with sequence length. The lack of fine-tuning requirements makes adoption straightforward, potentially enabling rapid ecosystem integration across open-source and commercial deployments.

• →OjaKV reduces KV-cache memory consumption through hybrid storage preserving important tokens while compressing intermediate ones.
• →Online subspace adaptation using Oja's algorithm enables the compression basis to adjust dynamically to evolving context.
• →The framework maintains or improves accuracy at high compression ratios without requiring model fine-tuning or retraining.
• →Compatibility with FlashAttention enables plug-and-play integration into existing inference pipelines.
• →Greatest performance improvements emerge on long-context benchmarks requiring complex reasoning, demonstrating value for production deployments.