Soft-Masked Diffusion Language Models

Researchers introduce soft-masking (SM), a novel approach for diffusion-based language models that improves upon traditional binary masked diffusion by blending mask token embeddings with predicted tokens. Testing on models up to 7B parameters shows consistent improvements in performance metrics and coding benchmarks.