Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

Researchers introduce Efficient-DLM, a framework for converting pretrained autoregressive language models into diffusion language models that enable parallel, non-autoregressive generation. The approach uses block-wise attention patterns and position-dependent masking to preserve model accuracy while achieving 4.5x higher throughput compared to existing models.

The research addresses a fundamental challenge in language model architecture: the tension between generation speed and accuracy. Traditional autoregressive models generate tokens sequentially, creating a computational bottleneck despite their strong task performance. Diffusion language models offer parallel generation but typically require training from scratch, losing the benefits of large-scale pretraining. Efficient-DLM bridges this gap through technical innovations that respect the weight distributions learned during autoregressive pretraining while enabling simultaneous token prediction.

The methodology builds on two key insights. First, block-wise attention—causal across blocks but bidirectional within blocks—preserves the inductive biases of autoregressive models better than fully bidirectional attention, while still enabling key-value caching for efficiency gains. Second, position-dependent masking during training better simulates the left-to-right token distribution observed during inference, reducing the training-test discrepancy that plagues existing diffusion approaches. These seemingly incremental refinements compound into substantial performance gains.

For the AI infrastructure market, this work has significant implications. The 4.5x throughput improvement over comparable models directly translates to reduced inference costs and latency, critical metrics for production deployments. The 8B variant outperforming larger models (Qwen3 4B) with superior accuracy demonstrates that architectural efficiency gains can rival scaling approaches. This challenges assumptions about the necessity of massive model sizes and suggests that optimization techniques applied to existing checkpoints may provide better cost-performance ratios than training larger models from scratch.

The research likely influences how organizations approach model deployment and fine-tuning strategies, particularly those balancing real-time performance requirements with accuracy constraints.

• →Efficient-DLM converts pretrained autoregressive models into faster diffusion models while maintaining accuracy
• →Block-wise attention pattern preserves pretrained weight distributions better than fully bidirectional approaches
• →Position-dependent masking strategy reduces training-test gap in token distribution behavior
• →8B variant achieves 4.5x higher throughput than Dream 7B with 5.4% better accuracy
• →Architectural optimization may provide better efficiency gains than pure scaling approaches