Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR

Researchers introduce REFT, a method that improves Reinforcement Learning with Verifiable Rewards (RLVR) by diversifying the first token generated after reasoning markers, addressing a previously overlooked bottleneck in rollout diversity. The technique achieves measurable improvements across multiple model sizes and difficulty levels without requiring changes to existing RLVR pipelines.

REFT targets a fundamental challenge in training reasoning models: generating diverse exploration paths while maintaining reliable correctness signals. Traditional RLVR approaches rely on temperature adjustments, prefix variations, or selection mechanisms to broaden rollout diversity, but this research identifies that the first token distribution exhibits a sharply peaked yet correctness-independent characteristic—meaning it can be manipulated for exploration without compromising learning signals. This distinction matters because the policy's initial token choice fundamentally shapes which reasoning paths emerge during rollout generation.

The method represents an incremental but meaningful advancement in the AI reasoning research space. By uniformly sampling from the policy's top-N first-token candidates and distributing rollouts evenly, REFT achieves improvements in Pass@1, Pass@8, and Pass@64 metrics across models ranging from 0.5B to 7B parameters. The lightweight implementation requires minimal pipeline modifications, suggesting practical adoption potential for teams already using RLVR frameworks.

For the broader AI development ecosystem, this research demonstrates how focused structural analysis can unlock performance gains without architectural overhauls. The consistent improvements across difficulty regimes and model scales indicate the technique's robustness. However, this remains specialized research relevant primarily to organizations training reasoning models with verifiable rewards—a growing but still niche focus within AI development. The work contributes to the ongoing effort to make AI systems more reliable through better training methodologies, supporting the longer-term trend toward more trustworthy AI reasoning.

• →REFT improves rollout diversity by focusing on first-token sampling after reasoning markers, addressing an overlooked structural bottleneck in RLVR training.
• →The method achieves measurable improvements across model sizes (0.5B-7B) and difficulty levels without modifying existing RLVR pipeline components.
• →First-token distributions are shown to be sharply peaked yet decoupled from correctness signals, enabling safe exploration exploitation.
• →The lightweight implementation suggests practical adoption potential for teams working with verifiable reward training approaches.
• →Results demonstrate consistent gains in Pass@1, Pass@8, and Pass@64 metrics, indicating robustness across different evaluation scales.