The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs

Research shows that large language models' performance on short tasks may underestimate their capabilities, as small improvements in single-step accuracy lead to exponential gains in handling longer tasks. The study reveals that larger models excel at execution over many steps, though they suffer from 'self-conditioning' where previous errors increase the likelihood of future mistakes, which can be mitigated through 'thinking' mechanisms.

• →Short-task benchmarks may create an illusion of diminishing returns in LLM scaling, masking exponential improvements in long-horizon task completion.
• →Larger models demonstrate significantly better execution capability across multiple turns even when smaller models achieve near-perfect single-turn accuracy.
• →Models exhibit self-conditioning behavior where previous errors in context increase the probability of making subsequent mistakes.
• →Self-conditioning effects persist despite model scaling but can be mitigated through thinking mechanisms during execution.
• →The research suggests continued scaling benefits for complex reasoning tasks that require extended execution sequences.