Quantifying Empirical Compute-Supervision Tradeoffs in RLVR

Researchers empirically tested whether increased compute can overcome imperfect verifier performance in reinforcement learning from verifiable rewards (RLVR), finding that verifier quality and training compute are not interchangeable. The study reveals that false negatives degrade model performance more severely than false positives, and compute scaling alone cannot close performance gaps caused by supervision noise.

This research challenges a foundational assumption in post-training language models: that sufficient computational resources can compensate for imperfect reward signals. The study systematically injected controlled noise into binary correctness signals while training Qwen2.5 models on mathematical reasoning tasks, varying compute budgets through rollout scaling. The findings contradict theoretical predictions that suggested verifier noise only affects learning speed, not final performance.

The work emerges from growing recognition that RLVR has become central to language model improvement, yet real-world verifiers frequently produce errors. Prior theoretical frameworks suggested this was merely an optimization problem—more training would eventually reach the same destination. This empirical work demonstrates otherwise, revealing a structural asymmetry: false negatives (rejecting correct outputs) degrade performance significantly faster than false positives (accepting incorrect outputs), suggesting asymmetric vulnerability in learning dynamics.

For AI developers and organizations scaling post-training pipelines, these findings have immediate implications. The result that verifier quality cannot be traded for compute fundamentally reshapes resource allocation strategies. Rather than purely scaling training infrastructure, teams must prioritize verifier accuracy and robustness as primary levers. This could redirect investment toward verifier development, multi-stage verification systems, and noise-robust training algorithms.

Looking forward, this opens research directions in verifier design optimization and noise-tolerant training methods. The asymmetric impact of error types suggests targeted interventions focusing on false-negative reduction could yield outsized returns compared to general compute increases. The findings also highlight potential limitations in current RLVR approaches as models scale, potentially informing architecture choices for frontier model development.

• →Verifier quality and training compute are not interchangeable—perfect verifier performance cannot be compensated through compute scaling alone.
• →False negatives degrade model performance more rapidly than false positives, revealing asymmetric vulnerability in reward signal quality.
• →Compute returns diminish sharply under verifier noise, suggesting verifier accuracy should be prioritized over pure computational scaling.
• →Current theoretical predictions about RLVR learning dynamics do not match empirical observations on realistic model scales.
• →Verifier robustness and design optimization emerge as critical but underexplored levers compared to standard compute expansion.