On the Optimal Reasoning Length for RL-Trained Language Models

Researchers studying reinforcement learning-trained language models discover that reasoning accuracy peaks at intermediate chain-of-thought lengths rather than improving monotonically with longer outputs. While sample accuracy declines beyond optimal length, the modal accuracy continues improving, suggesting longer reasoning produces both more correct and more variable outputs.

This research addresses a fundamental inefficiency in modern AI systems: RL-trained language models tend to generate excessively long reasoning chains, inflating computational costs without proportional accuracy gains. The study's core finding—that accuracy follows an inverted-U curve with output length—challenges assumptions underlying many length-control approaches and provides empirical grounding for optimizing inference efficiency.

The distinction between sample accuracy and modal accuracy proves crucial. While individual reasoning paths show non-monotonic performance (peaking then declining), the most likely or central tendency continues improving with length. This suggests the problem isn't fundamental reasoning capacity but rather increased output variance at longer lengths, where models explore more diverse reasoning paths with mixed quality.

For practitioners deploying these models, this research enables more informed cost-benefit calculations. Rather than simply constraining length to reduce computation, developers can now target the optimal sweet spot where accuracy peaks while minimizing wasted tokens. The findings apply across multiple domains—mathematical reasoning and code generation—suggesting broad applicability.

The implications extend beyond efficiency. Understanding this relationship helps explain why longer reasoning doesn't always help and informs better training strategies. As AI systems scale and inference costs climb, identifying optimal reasoning lengths becomes economically significant. Future work should explore whether this pattern holds across different RL training approaches and whether guided sampling can improve modal accuracy while maintaining output diversity.

• →Accuracy in RL-trained models peaks at intermediate reasoning lengths, not at maximum length
• →Modal accuracy continues improving with length even when sample accuracy declines, indicating increased variance
• →Length-control methods need refinement to target optimal accuracy points rather than arbitrary constraints
• →The findings apply consistently across mathematical reasoning and code generation tasks
• →Optimizing reasoning length offers significant computational cost reduction without accuracy sacrifice