GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

Researchers propose Guided Denoiser Self-Distillation (GDSD), a new reinforcement learning method for diffusion language models that eliminates the need for evidence lower bound approximations, achieving up to 19.6% performance improvements over existing approaches on planning, math, and coding tasks.

The research addresses a fundamental technical challenge in applying reinforcement learning to diffusion-based language models. Traditional RL approaches struggle with diffusion models because calculating policy likelihood is computationally intractable, leading researchers to use evidence lower bounds (ELBO) as approximations. However, this workaround creates a training-inference mismatch that degrades performance, as the model optimizes for a proxy objective rather than actual likelihood.

GDSD bypasses this problem through direct self-distillation, where a teacher model derived from optimal reverse-KL regularized RL guides the denoiser's learning. This approach reduces the complex RL problem to a simpler distillation task without likelihood surrogates. The method represents a conceptual shift in how researchers approach RL for diffusion models, moving away from approximation-based methods toward direct optimization.

The empirical results are substantial: GDSD consistently outperforms ELBO-based methods across multiple benchmarks with more stable training dynamics. The 19.6% accuracy improvement on certain tasks suggests meaningful practical gains. This matters for AI developers building diffusion-based language models, as it provides a more reliable optimization path that doesn't rely on mathematical approximations that can accumulate errors during training.

For the broader AI field, this work indicates that diffusion models—an increasingly popular architecture class—can achieve better performance through improved training methodology rather than architectural changes. The open-sourced code enables rapid adoption, potentially influencing how future diffusion model training is approached across research and industry applications.

• →GDSD eliminates the evidence lower bound approximation used in prior RL methods for diffusion language models
• →Direct denoiser self-distillation achieves up to 19.6% accuracy improvements on planning, math, and coding tasks
• →The method provides more stable training dynamics with reduced bias compared to ELBO-based approaches
• →GDSD represents a shift from likelihood surrogates to direct optimization in diffusion model training
• →Code is publicly available, enabling adoption across research and production AI systems