PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

Researchers propose Predictive Routing Replay (PR2), a technique to stabilize reinforcement learning training on Mixture of Experts LLMs by predicting router evolution and reducing the mismatch between rollout and training phases. The method addresses router drift—a critical instability source in MoE-based models undergoing RL fine-tuning—through lightweight prediction mechanisms that anticipate expert activation changes.

Training Mixture of Experts language models with reinforcement learning presents a significant technical challenge that has limited the scalability and stability of these increasingly popular architectures. The core problem stems from router drift: as models update during training, the routing decisions that determine which experts process which tokens can shift dramatically, creating inconsistencies between the rollout phase (where training data is generated) and the training phase (where model weights are updated). This mismatch destabilizes importance sampling weights in PPO-style algorithms, making training unpredictable and inefficient.

PR2 advances beyond prior routing replay approaches by introducing a learned evolution predictor that forecasts how routing patterns will change in the near term. Rather than freezing routes statically, the method anticipates router behavior post-update, enabling more informed expert selection during rollouts. This predictive component directly targets the staleness problem inherent in simpler frozen-routing strategies, which ignore the dynamic nature of model learning.

The implications for AI development are substantial. MoE models represent the frontier of efficient scaling—they reduce computational overhead while maintaining capability. However, instability during RL fine-tuning limits their practical deployment for reasoning tasks and preference alignment. Stabilizing this training regime unlocks a significant bottleneck in scaling aligned AI systems. Better RL training stability on MoE models accelerates deployment of more capable, cost-effective systems across research and production environments.

The theoretical backing and empirical validation across reasoning benchmarks suggest this approach has genuine technical merit. Success here could become standard practice in MoE model optimization, influencing how AI labs train next-generation reasoning models and potentially affecting compute efficiency across the industry.

• →PR2 introduces predictive routing to solve router drift instability in MoE-based LLM reinforcement learning
• →The method uses lightweight evolution predictors to anticipate expert activation changes within training trajectories
• →Router staleness from frozen routes is mitigated by predicting short-horizon router evolution during updates
• →Empirical results demonstrate improved training stability and stronger performance on reasoning benchmarks
• →The approach could become standard practice for stabilizing RL training on increasingly popular MoE architectures