The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL

Researchers propose Optimal Token Baseline (OTB), a new variance reduction technique for reinforcement learning in large language models that addresses training instability in long-horizon tasks. The method reduces token consumption by over 65% while maintaining performance equivalent to models using 8x larger batch sizes, offering significant efficiency gains for LLM-RL training.

This research addresses a fundamental challenge in reinforcement learning applied to large language models: controlling gradient variance during training on extended sequences. The exploding variance problem has historically limited the scalability and stability of RL approaches for LLMs, forcing researchers to use increasingly large batch sizes and computational resources to achieve convergence. The proposed Optimal Token Baseline derives mathematically optimal weighting for gradient updates based on cumulative gradient norms, establishing a principled foundation for variance reduction that previous approaches lacked.

The innovation centers on the Logit-Gradient Proxy, which enables efficient approximation of gradient norms using only forward-pass probabilities rather than expensive gradient computations. This computational efficiency breakthrough allows practitioners to achieve training stability with batch size N=4 that previously required N=32, representing an 8-fold reduction in required samples. The 65% token consumption reduction across reasoning and tool-integrated tasks demonstrates practical applicability beyond toy problems.

For the AI infrastructure ecosystem, this development directly impacts training costs and accessibility. Organizations building production LLM-RL systems—including those developing AI agents, reasoning models, and code generation systems—can substantially reduce computational requirements and carbon footprint. This democratizes advanced RL training methods to researchers and companies with more limited resources. The efficiency gains compound when applied to the iterative nature of RL training, where multiple rounds of refinement are standard practice.

Future developments may involve integrating OTB into existing RL frameworks and exploring whether similar token-level optimization principles apply to other training instabilities in large models. The technique's generalizability across different model architectures and task domains remains an important validation frontier.

• →Optimal Token Baseline reduces token consumption by 65% while matching performance of 8x larger batch sizes in LLM-RL training
• →Gradient weighting based on cumulative gradient norms provides mathematically optimal variance reduction for long-horizon tasks
• →Logit-Gradient Proxy enables efficient implementation using only forward-pass probabilities, eliminating expensive gradient computations
• →Technique addresses sequence heterogeneity overlooked by traditional value-based and group-based baseline methods
• →Efficiency gains directly reduce computational costs and accessibility barriers for LLM-RL applications in production environments