Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards

Researchers propose ACOER, a novel training method that stabilizes efficiency optimization in large language models by applying length penalties only to correct answers, avoiding the reward collapse problems that plague existing approaches. The technique achieves 60% token reduction while maintaining or improving reasoning accuracy across mathematical benchmarks.

The challenge of training reasoning models to be both accurate and efficient has exposed fundamental vulnerabilities in current optimization frameworks. GRPO's group normalization mechanism creates an unexpected failure mode when length penalties apply uniformly across correct and incorrect responses—penalizing incorrect answers causes divergent advantage signals that destabilize training. This structural issue represents a critical blind spot in reinforcement learning design for language models, where intuitive penalty schemes produce counterintuitive collapse outcomes.

This research builds on years of work attempting to reduce model verbosity without sacrificing reasoning quality. Prior efforts struggled because efficiency rewards and correctness rewards operate on different optimization gradients, creating tension that existing frameworks couldn't reconcile. The breakthrough insight—isolating efficiency incentives to only correct completions—transforms the problem from adversarial to complementary, allowing both objectives to reinforce rather than undermine each other.

For developers building production reasoning systems, this work directly impacts inference costs and latency. Token reduction of 60% translates to substantial computational savings and faster response times, making advanced reasoning capabilities economically viable for cost-sensitive applications. The stability guarantees embedded in ACOER's design reduce the risk of training failures that can waste weeks of GPU resources.

The methodology opens pathways for scaling reasoning capabilities more efficiently across domains beyond mathematics. Future work likely explores ACOER application to code generation, scientific reasoning, and multi-step planning tasks where verbosity significantly impacts deployment economics. The control-loop penalty adjustments suggest adaptive optimization could extend beyond efficiency to other constrained objectives.

• →Applying length penalties to incorrect answers causes structural reward collapse in GRPO due to group normalization divergence
• →ACOER restricts brevity rewards to correct completions, eliminating the structural failure while preventing over-compression via dynamic budgeting
• →The method achieves 60% token reduction while maintaining or improving accuracy across mathematical reasoning benchmarks
• →Efficient reasoning models reduce inference costs and latency critical for production deployment of advanced language model capabilities
• →The approach generalizes beyond mathematics to code generation and multi-step reasoning tasks with verbosity constraints