ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning

Researchers introduce ReasonBENCH, a comprehensive benchmark revealing that LLM reasoning systems exhibit significant performance variance across repeated executions, with the best-performing strategy winning only 77% of head-to-head comparisons. The study demonstrates that this instability is structured rather than random, challenging the validity of single-run benchmark scores as reliable indicators of model quality.

The ReasonBENCH study addresses a fundamental problem in AI evaluation: the assumption that a single benchmark score represents a model's true capability. By running 30 independent trials across multiple models and strategies, researchers found that identical configurations produce meaningfully different outputs and computational costs, even under deterministic settings. This discovery has profound implications for how the AI community validates and compares systems.

The research distinguishes between two types of variance—Global Noise affecting cross-benchmark consistency and Run Noise capturing within-benchmark variability—revealing that instability correlates with strategy architecture rather than random fluctuations. This structured nature suggests the problem is addressable through design improvements rather than inevitable. The hierarchical decomposition shows that three-quarters of variance stems from benchmark, system, and item structure, with a stubborn residual that single evaluations mask entirely.

For practitioners and investors, these findings challenge widespread confidence in published benchmark rankings. A model ranking first in one evaluation could legitimately rank second in another, creating decision-making ambiguity. The asymmetric cost-quality decoupling is particularly notable: cheaper inference methods show structural resilience to joint failure, while expensive approaches remain vulnerable regardless of accuracy. This reframes cost-benefit calculations in production deployments.

The study establishes distribution-aware evaluation as necessary standard practice rather than optional rigor. Organizations should demand confidence intervals alongside point estimates and conduct multi-trial assessments before deployment decisions. Future benchmarking frameworks will likely incorporate ReasonBENCH's methodology, shifting the industry toward more statistically robust comparisons.

• →LLM reasoning benchmark scores are unreliable when reported as single numbers, with top strategies winning only 77% of head-to-head matchups against competitors
• →Performance variance is structured and predictable based on strategy architecture, enabling targeted stability improvements rather than treating instability as random noise
• →Three-quarters of score variance comes from benchmark and system structure, while a persistent residual remains invisible to single-run evaluations
• →Cost and quality decouple asymmetrically: budget-conscious methods are inherently stable against joint failure while expensive methods remain exposed
• →Distribution-aware evaluation methodologies should become standard practice to replace unreliable point-estimate benchmarking