Unlocking Prompt Infilling Capability for Diffusion Language Models

Researchers have developed a method to unlock prompt infilling capabilities in masked diffusion language models by extending full-sequence masking during supervised fine-tuning, rather than the conventional response-only masking. This breakthrough enables models to automatically generate effective prompts that match or exceed manually designed templates, suggesting training practices rather than architectural limitations were the primary constraint.

• →Masked diffusion language models can now infill prompts by using full-sequence masking during supervised fine-tuning instead of response-only masking.
• →Model-generated prompts perform as well as or better than manually designed prompt templates.
• →The improved prompts transfer effectively across different language models.
• →Training methodology, not model architecture, was the main bottleneck preventing this capability.
• →This technique is complementary to existing prompt optimization methods.