Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training

Researchers present a study optimizing reinforcement learning for autoregressive text-to-image generation by analyzing how different divergence measures affect policy alignment. Using JS divergence within the GRPO framework, they demonstrate improved performance across evaluation metrics while preserving generation diversity on LlamaGen and Janus-7B models.

This research addresses a critical technical challenge in aligning autoregressive text-to-image models with human preferences through reinforcement learning. The study systematically examines how reference-policy divergence—the measure controlling how far a model can deviate from its baseline—impacts optimization outcomes, a factor largely overlooked in previous approaches. By applying f-divergence theory to GRPO-style training, the researchers demonstrate that JS divergence offers superior trade-offs compared to existing methods like forward and reverse KL divergence.

The work builds on the rapidly advancing field of generative AI, where aligning model outputs with human preferences remains computationally expensive and theoretically complex. Recent breakthroughs in autoregressive T2I generation have created new possibilities for creative applications, yet maintaining quality while encouraging diverse outputs remains challenging. This research contributes to understanding the mathematical foundations underlying policy optimization in generative models.

For developers and AI researchers, the findings provide actionable guidance on divergence selection during model training, potentially reducing computational overhead and improving output quality. The open-source code release enables broader adoption and validation across different architectures. The demonstrated success on both LlamaGen and Janus-7B suggests the approach generalizes effectively across model families.

Looking forward, this work may influence how future autoregressive models are trained, particularly for applications requiring balanced performance and diversity. The theoretical framework could extend to other domains beyond image generation, establishing principles applicable to multimodal alignment problems.

• →JS divergence provides superior trade-offs between policy alignment and generation diversity in autoregressive T2I training
• →Different f-divergence measures reshape token-level updates distinctly, requiring careful selection during GRPO optimization
• →The approach achieves strongest performance across multiple evaluation metrics while maintaining favorable diversity on LlamaGen and Janus-7B
• →Open-source implementation enables broader adoption and validation by the research community
• →Findings suggest principles may generalize to other multimodal alignment and policy optimization problems