Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?

Researchers identify why Diffusion Language Models (DLMs) struggle with parallel token generation, finding that training data structure forces autoregressive-like behavior. They propose NAP, a data-centric approach using multiple independent reasoning trajectories that improves parallel decoding performance on math benchmarks.

• →Diffusion Language Models often converge to sequential decoding despite being designed for parallel generation.
• →The mismatch between training data structure and parallel objectives causes autoregressive-like behavior.
• →NAP approach uses multiple independent reasoning trajectories instead of sequential chain-of-thought data.
• →Performance gains increase with higher levels of parallelism in the proposed method.
• →Data-centric solutions may be key to achieving truly non-autoregressive language generation.