Enabling KV Caching of Shared Prefix for Diffusion Language Models

Researchers introduce bicache, a novel KV caching technique that enables efficient serving of diffusion language models (DLMs) with shared prefixes. Unlike traditional LLMs, DLMs use bidirectional attention, which invalidates conventional caching methods and causes accuracy collapse. Bicache dynamically identifies safe layer depths for prefix reuse, achieving 36-98% throughput improvements.

The emergence of diffusion language models represents a fundamental shift in neural network architecture, moving away from the unidirectional attention mechanisms that have dominated since the transformer's introduction. This architectural change creates a critical technical problem: existing optimization techniques developed for LLMs become counterproductive in DLMs because bidirectional attention means cached key-value pairs become stale whenever the model processes new tokens.

The research addresses a bottleneck in production ML systems. High-throughput LLM serving relies heavily on KV caching to avoid recomputing attention for repeated token sequences—a critical optimization for cost-effective deployment. When DLMs emerged as a promising alternative architecture, this core optimization technique broke entirely, forcing practitioners to choose between model performance and computational efficiency. The bicache solution observes that while deep layers require recalculation, shallow layers maintain stable KV representations across bidirectional passes, enabling selective caching based on prefix length.

For the AI infrastructure industry, this development unlocks practical DLM deployment at scale. Organizations investing in DLM research can now expect comparable throughput to traditional LLMs, reducing the deployment friction that might have otherwise favored established LLM architectures. The 36-98% throughput gains translate directly to reduced inference costs and improved service latency, making DLM adoption economically viable.

The significance extends beyond immediate performance metrics. This work demonstrates how architectural innovations in deep learning often require complementary systems-level innovations to achieve practical viability. As alternative attention mechanisms proliferate, similar caching strategies will likely prove essential across emerging model families.

• →Bidirectional attention in DLMs invalidates standard LLM caching techniques, causing near-zero accuracy without new solutions
• →Bicache achieves 36-98% throughput improvements by dynamically identifying safe shallow layers for KV prefix reuse
• →Shallow layer KVs remain stable across bidirectional passes, enabling selective caching without accuracy degradation
• →Safe caching depth varies dynamically based on shared prefix token fraction in each request batch
• →Production DLM serving now becomes economically competitive with traditional LLM deployment