Align and Filter: Improving Performance in Asynchronous On-Policy RL

Researchers propose a new method called total Variation-based Advantage aligned Constrained policy Optimization to address policy lag issues in distributed reinforcement learning systems. The approach aims to improve performance when scaling on-policy learning algorithms by mitigating the mismatch between behavior and learning policies during high-frequency updates.

• →Policy lag occurs when there's a mismatch between the behavior policy generating data and the learning policy being updated in distributed RL systems.
• →Both distributed training and increased gradient update frequency can worsen policy lag problems.
• →The proposed method shows better robustness to policy lag in classic RL tasks and LLM math reasoning applications.
• →This research addresses a key scaling challenge for on-policy reinforcement learning algorithms.
• →The findings have implications for training large language models using reinforcement learning techniques.