Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

Researchers introduce On-Policy Diffusion Language Models (OPDLM), a technique that converts autoregressive language models into diffusion models using 15-7,000x fewer training tokens. The method addresses fundamental efficiency problems by eliminating train-inference mismatches and preserving knowledge from the original model through on-policy distillation.

The paper addresses a critical inefficiency in current diffusion language model research. Converting autoregressive models to diffusion models typically requires prohibitive computational costs and suffers from two distinct problems: knowledge loss during objective transition and misalignment between training (random masking) and inference (confidence-based decoding) processes. OPDLM solves these by having a bidirectional student model generate its own trajectories while receiving guidance from the frozen teacher (original autoregressive model), creating a self-reinforcing loop that maintains knowledge while adapting to realistic inference conditions.

This advancement emerges from growing recognition that diffusion models, while promising for language generation, face scalability challenges compared to autoregressive alternatives. The research landscape has increasingly focused on transformation methods rather than pretraining from scratch, yet prior approaches incurred substantial distribution shifts. By framing the problem as post-training rather than pretraining, OPDLM sidesteps the most computationally expensive phase of model development.

The implications extend beyond academic interest. The dramatic reduction in required training tokens—potentially 7,000x fewer—could democratize diffusion language model development, making it accessible to organizations without massive compute resources. This efficiency gain particularly matters for fine-tuning and model adaptation across specialized domains. For the broader AI industry, efficient transformation techniques reduce barriers to exploring diffusion-based architectures, potentially accelerating research into parallel decoding and other inference improvements that autoregressive models struggle with. The work positions diffusion models as practical alternatives rather than theoretical curiosities.

• →OPDLM achieves 15-7,000x reduction in training tokens required for autoregressive-to-diffusion model conversion
• →On-policy distillation eliminates train-inference mismatch by training on realistic decoding trajectories rather than random masks
• →Knowledge retention from original autoregressive models improves performance while reducing computational overhead
• →Technique repositions diffusion model development as efficient post-training rather than expensive pretraining
• →Method enables broader exploration of diffusion architectures for parallel decoding and inference improvements