Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers

Researchers introduce RL^V, a reinforcement learning method that unifies LLM reasoners with generative verifiers to improve test-time compute scaling. The approach achieves over 20% accuracy gains on MATH benchmarks and enables 8-32x more efficient test-time scaling compared to existing RL methods by preserving and leveraging learned value functions.

The research addresses a fundamental limitation in current large language model training approaches. Existing RL methods like GRPO and Leave-one-out PPO discard learned value functions in favor of empirical return estimates, creating inefficiencies when deploying models that require parallel test-time computation. RL^V solves this by jointly training the language model as both a reasoner and a generative verifier using data already produced during RL training, eliminating the false choice between reasoning performance and verification capability.

This work emerges from the broader trend of test-time scaling in AI, where models benefit from additional computational resources during inference rather than just training. As organizations increasingly plan deployments with parallel compute availability, training methodologies must evolve to capitalize on these capabilities. The co-training principle underlying RL^V represents a shift toward more holistic model development that aligns training objectives with actual deployment constraints.

The commercial implications are substantial. For AI developers and companies building reasoning systems, RL^V offers significant efficiency gains—8-32x improvements in test-time scaling translate directly to reduced computational costs or improved performance within budget constraints. The method's strong generalization across easy-to-hard and out-of-domain tasks suggests robustness, while the 1.2-1.6x performance boost when combined with advanced reasoning models indicates compatibility with next-generation approaches.

The methodology's focus on co-optimization hints at future directions where training design explicitly incorporates deployment realities. Organizations developing LLM-based verification and reasoning systems should monitor adoption rates and performance benchmarks to assess whether this represents the emerging standard for test-time scaling.

• →RL^V unifies LLM reasoners with generative verifiers, achieving 20%+ accuracy improvements on MATH benchmarks through joint training
• →Method enables 8-32x more efficient test-time compute scaling by preserving value functions that existing RL methods discard
• →Training approach adds verification capabilities without significant computational overhead during model development
• →Strong generalization across difficulty levels and out-of-domain tasks demonstrates robustness of the co-training principle
• →1.2-1.6x performance gains when combined with long-context reasoning models suggests compatibility with next-generation LLM architectures