Diffusion Reinforcement Learning via Centered Reward Distillation

Researchers present Centered Reward Distillation (CRD), a new reinforcement learning framework for fine-tuning diffusion models that addresses brittleness issues in existing methods. The approach uses within-prompt centering and drift control techniques to achieve state-of-the-art performance in text-to-image generation while reducing reward hacking and convergence issues.

• →CRD framework solves brittleness problems in diffusion reinforcement learning through KL-regularized reward maximization.
• →Within-prompt centering technique cancels out intractable normalizing constants to create a well-posed reward-matching objective.
• →The method introduces three drift control techniques: decoupled sampling, KL anchoring, and reward-adaptive KL strength.
• →Experiments show competitive SOTA results with faster convergence and reduced reward hacking on text-to-image tasks.
• →The approach addresses key weaknesses in diffusion models including prompt fidelity, compositional correctness, and text rendering.