Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models

Researchers propose ADAS, a training-free reranking algorithm that improves parallel token decoding in masked diffusion language models by using attention weights as soft penalties to avoid committing to correlated predictions simultaneously. The method achieves 9-10 percentage point improvements on benchmarks like GSM8K and HumanEval with minimal computational overhead, advancing the efficiency of faster language model inference.

Masked diffusion language models represent an emerging approach to accelerate transformer inference by predicting multiple tokens in parallel rather than sequentially. While this parallelism offers theoretical speedups, the core challenge is determining which tokens are safe to commit together—a problem the new ADAS framework directly addresses. The research demonstrates that naive token-selection methods fail to account for statistical dependencies between predictions, leading to quality degradation when multiple uncertain, correlated predictions are generated simultaneously.

This work builds on recent progress in non-autoregressive and iterative decoding methods that challenge the sequential token-by-token generation paradigm dominating modern LLMs. Prior samplers like Top-k and Fast-dLLM control only the quantity of tokens to reveal per iteration, ignoring interaction effects. ADAS introduces a soft attention-based penalty mechanism that discounts candidate tokens when they attend strongly to already-selected positions with uncertain predictions, avoiding the rigidity of hard constraint-based approaches used in graph-constrained methods.

The empirical validation across 8B and 7B parameter models on mathematical reasoning (MATH500, GSM8K) and code generation tasks (HumanEval, MBPP) shows consistent improvements in low-NFE (Number of Function Evaluations) regimes, where parallel decoding provides maximum benefit. The 3.1% runtime overhead is negligible relative to the quality gains, making ADAS immediately practical for production systems. For AI researchers and practitioners, this represents a modular, training-free technique that enhances existing samplers without requiring model retraining. The approach signals growing maturity in efficient inference methods that could reshape deployment economics for large language models.

• →ADAS improves parallel masked diffusion decoding by using attention weights as soft penalties to avoid selecting correlated uncertain predictions together
• →The method achieves 9-10 percentage point improvements on benchmarks like GSM8K and HumanEval without requiring model retraining
• →ADAS is a training-free, modular approach that enhances existing samplers like Top-k and Fast-dLLM with only 3.1% runtime overhead
• →The research addresses a fundamental challenge in parallel decoding: determining which tokens can safely be predicted simultaneously
• →Soft attention-based penalties outperform hard constraint-based methods while maintaining computational efficiency