Flow-OPD: On-Policy Distillation for Flow Matching Models

Researchers introduce Flow-OPD, a post-training framework that applies on-policy distillation to Flow Matching text-to-image models, addressing reward sparsity and gradient interference problems. Built on Stable Diffusion 3.5 Medium, the method achieves significant performance gains—GenEval scores improve from 63 to 92 and OCR accuracy from 59 to 94—while maintaining image quality and surpassing individual teacher models.

Flow-OPD represents a meaningful advancement in aligning text-to-image generative models with multiple, often conflicting objectives. The framework tackles a fundamental challenge in multi-task reinforcement learning: when models optimize for several rewards simultaneously, performance degrades across metrics due to competing gradients and sparse feedback signals. By adopting a two-stage approach—first training specialized expert models independently, then consolidating their knowledge into a single student—the researchers circumvent the 'seesaw effect' where improvements in one area cause regressions elsewhere.

The introduction of Manifold Anchor Regularization provides a stabilizing mechanism that grounds generation quality to a high-fidelity reference manifold, preventing the aesthetic degradation typically observed in purely RL-driven model alignment. This design choice reflects lessons learned from large language model development, where similar distillation techniques have proven effective.

The performance improvements are substantial. Raising GenEval scores by 29 points and OCR accuracy by 35 points on a model already trained on large-scale data demonstrates that the alignment methodology itself contributes significant value. The 'teacher-surpassing' effect—where the student outperforms its individual teachers—indicates that the orchestration of multiple expertise sources creates emergent capabilities.

For the generative AI industry, Flow-OPD establishes a scalable post-training paradigm that other teams can adapt. The framework's applicability to Stable Diffusion 3.5 suggests compatibility with widely-deployed open models. This work could accelerate development of more capable text-to-image systems by providing a principled approach to multi-objective alignment without sacrificing fidelity or introducing reward hacking.

• →Flow-OPD achieves 29-point GenEval improvement and 35-point OCR accuracy gain through on-policy distillation of specialized teacher models
• →Two-stage alignment strategy isolates single-task optimization before consolidating heterogeneous objectives, eliminating competing metric degradation
• →Manifold Anchor Regularization prevents aesthetic quality loss commonly observed in pure RL-driven alignment by anchoring to high-quality reference distributions
• →Student model exhibits 'teacher-surpassing' effect, outperforming individual expert teachers and GRPO baseline by roughly 10 points overall
• →Framework demonstrates scalability and applicability to widely-deployed models like Stable Diffusion, enabling broader adoption in generative AI development