Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow

Researchers evaluated eight large Masked Diffusion Language Models (up to 100B parameters) and found they still underperform comparable autoregressive models despite promises of parallel token generation. The study reveals MDLMs exhibit task-dependent decoding behavior and propose a Generate-then-Edit paradigm to improve performance while maintaining parallel processing efficiency.

Masked Diffusion Language Models represent a fundamental alternative to autoregressive decoding, theoretically enabling tokens to be generated simultaneously rather than sequentially. This research quantifies the performance gap between this promising paradigm and current state-of-the-art autoregressive models across 58 benchmarks, revealing that the theoretical advantages have not yet translated into practical superiority. The core limitation stems from parallel probabilistic modeling's inherent weakness in capturing inter-token dependencies—the sequential relationships that autoregressive models excel at modeling.

The adaptive decoding behavior findings demonstrate that MDLMs are not uniformly parallel across all tasks. Instead, they dynamically adjust their generation strategy based on task requirements, reasoning stages, and solution correctness. This suggests the models implicitly learn which tokens can be predicted reliably in parallel and which require sequential refinement. Notably, MDLMs show genuine advantages on backward-information tasks like Sudoku, where filling easier positions first represents a more natural solving strategy than arbitrary sequential ordering.

The proposed Generate-then-Edit paradigm offers a pragmatic path forward. Rather than attempting pure parallel generation, this approach leverages parallel decoding's efficiency while recovering some dependency modeling through refinement. This hybrid strategy could make MDLMs competitive with autoregressive models while preserving computational gains for latency-sensitive applications. The research implies that scaling alone cannot solve the dependency problem; architectural innovations are necessary. For AI practitioners developing language models, this work suggests that task-aware decoding strategies and post-generation refinement may be critical components of production-grade diffusion language models.

• →Current MDLMs significantly underperform autoregressive models despite theoretical advantages in parallelism
• →Parallel probabilistic modeling weakens inter-token dependencies, the primary performance bottleneck
• →MDLMs exhibit adaptive decoding behavior that varies by task domain, reasoning stage, and correctness
• →MDLMs show genuine advantages on tasks requiring backward information like constraint satisfaction problems
• →Generate-then-Edit paradigm can recover dependency modeling while maintaining parallel decoding efficiency