Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning

Researchers propose Coupled Policy Optimization (CPO), a new reinforcement learning method that regulates policy diversity through KL constraints to improve exploration efficiency in large-scale parallel environments. The method outperforms existing baselines like PPO and SAPG across multiple tasks, demonstrating that controlled diverse exploration is key to stable and sample-efficient learning.

• →Coupled Policy Optimization uses KL constraints to regulate diversity between policies in ensemble learning methods.
• →The method outperforms strong baselines including SAPG, PBT, and PPO in both sample efficiency and final performance.
• →Excessive exploration can reduce learning quality and training stability, making regulation crucial.
• →Follower policies naturally distribute around leader policies, creating structured exploratory behavior.
• →The research addresses scaling reinforcement learning to tens of thousands of parallel environments.