DARE: Diffusion Language Model Activation Reuse for Efficient Inference

Researchers introduce DARE, a technique that reduces computational redundancy in Diffusion Language Models by reusing cached attention activations across tokens. The method achieves up to 1.20x per-layer latency improvements while maintaining generation quality, addressing efficiency gaps between diffusion-based and auto-regressive language models.

Diffusion Language Models represent an emerging architecture that processes text generation differently from dominant auto-regressive models, theoretically enabling faster parallel computation. However, open-source implementations have struggled with efficiency and output quality, limiting adoption. The DARE research identifies a previously underexploited inefficiency: token-wise redundancy in bi-directional self-attention mechanisms, where activation patterns repeat across tokens in predictable ways.

This discovery builds on growing recognition that transformer models contain substantial computational waste. Prior work has explored activation pruning and caching strategies, but DARE uniquely applies these insights to diffusion architectures. The two-pronged approach—reusing key-value caches (DARE-KV) and output activations (DARE-O)—targets complementary redundancy sources while maintaining model interpretability without requiring retraining.

For the broader AI infrastructure landscape, optimizing diffusion models carries significant implications. As organizations explore alternative architectures to auto-regressive transformers for specific use cases, efficiency improvements directly impact deployment costs and latency-sensitive applications. The ability to reuse 87% of attention activations with minimal quality degradation suggests substantial headroom for practical systems. The reported performance drops of 2.0% and 1.2% respectively remain within acceptable tolerances for many production scenarios.

The additive compatibility with existing optimization techniques like prefix caching amplifies DARE's practical value. As open-source diffusion models mature, such efficiency gains become prerequisites for competitive deployment. This research establishes a template for hardware-agnostic optimization strategies that could accelerate diffusion model adoption in resource-constrained environments, from edge devices to cost-conscious cloud deployments.

• →DARE reduces diffusion language model latency by up to 1.20x per-layer through attention activation reuse without model retraining.
• →Token-wise redundancy in self-attention enables reuse of up to 87% of activations with only 1.2-2.0% quality degradation.
• →The technique combines DARE-KV for key-value caching and DARE-O for output reuse as complementary optimization mechanisms.
• →Compatibility with existing optimization methods like prefix caching provides multiplicative efficiency improvements for production systems.
• →Results demonstrate diffusion-based LLMs can achieve computational parity with auto-regressive models on efficiency metrics.