Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment

Researchers introduce REPR-ALIGN, a method that converts autoregressive language models into diffusion language models by aligning their internal representations rather than retraining from scratch. The approach achieves up to 4x training acceleration and demonstrates that semantic structures learned through next-token prediction can transfer across different generation orders.

This research addresses a fundamental efficiency challenge in adapting existing language models to new generation paradigms. Rather than discarding the learned representations from autoregressive pretraining, the REPR-ALIGN method preserves this semantic structure through representation alignment, suggesting that generation order is a learnable mechanism separate from core linguistic understanding. This distinction represents a conceptual shift in how researchers think about model conversion—treating it as pathway retraining rather than complete representation relearning.

The motivation emerges from recent advances showing diffusion language models offer complementary strengths to autoregressive models, particularly for non-sequential generation and bidirectional editing tasks. Previous conversion methods required continued denoising training with objective modifications and attention-level adjustments, creating significant computational overhead. By using cosine similarity alignment between frozen pretrained representations and the diffusion model's hidden states at every layer, the approach achieves substantial training acceleration without architectural changes or adapter modules.

For the AI development community, this work has practical implications for model efficiency and accessibility. The 4x training speedup in certain settings reduces computational costs and democratizes access to multimodal generation capabilities. The technique proves particularly valuable in low-data regimes, enabling smaller organizations to adapt powerful pretrained models without prohibitive resource expenditure. The simple implementation—requiring only attention mask modifications—lowers the barrier to experimentation.

Looking ahead, this represents part of a broader trend toward representation-aware model adaptation techniques. Future research may extend these principles to cross-architecture conversion, multi-modal alignment, or domain-specific fine-tuning. The open-source code release signals community adoption potential, likely inspiring follow-up work on representation transfer across other model types and generation mechanisms.

• →REPR-ALIGN enables 4x training acceleration when converting autoregressive models to diffusion models through representation alignment
• →Semantic structures from next-token prediction transfer effectively across different generation orders without complete retraining
• →The method requires no adapter modules or architectural changes beyond attention masking, simplifying implementation
• →Representation alignment proves especially effective in low-data regimes, improving efficiency for resource-constrained scenarios
• →The technique reframes model conversion as learning new decoding paths rather than learning entirely new language representations