Generalized Discrete Diffusion with Self-Correction

Researchers propose Self-Correcting Discrete Diffusion (SCDD), a new AI model that improves upon existing discrete diffusion models by reformulating self-correction with explicit state transitions. The method enables more efficient parallel decoding while maintaining generation quality, demonstrating improvements at GPT-2 scale.

• →SCDD introduces a pretraining-based self-correction approach that learns directly in discrete time with explicit state transitions.
• →The framework simplifies training by eliminating redundant remasking steps and using exclusively uniform transitions.
• →Experiments show the method enables more efficient parallel decoding while preserving generation quality at GPT-2 scale.
• →The approach addresses limitations of prior work including poor generalization and impaired reasoning performance.
• →SCDD improves upon GIDD's continuous interpolation-based pipeline which had opaque interactions and complex hyperparameter tuning.