Position: State-of-the-Art Claims Require State-of-the-Art Evidence

Researchers identify a widespread gap between State-of-the-Art claims in AI/ML research and the evidence supporting them. Analysis of ten major benchmarks reveals that marginal improvements in aggregate scores often mask fragility, with gains driven by outlier datasets rather than meaningful superiority across tasks.

The AI research community faces a credibility problem rooted in how performance claims are validated. When papers declare SOTA status based on benchmark leaderboards, they implicitly promise superiority across diverse tasks—yet the evidence frequently falls short. The study examines this gap by analyzing top-model comparisons across ten cross-domain benchmarks, finding that over half fail basic tests of true superiority: meaningful effect size, consistency across tasks, or robustness when datasets are removed. This reveals a systematic disconnect between claim strength and empirical support. The root cause stems from how aggregate benchmarking works. A model achieving the highest average score across many tasks signals statistical top-ranking, not meaningful dominance. When performance differences narrow, especially in competitive leaderboards, rankings become sensitive to outlier datasets—tasks where one model vastly outperforms others. Remove those outliers, and the claimed superiority evaporates. This pattern persists despite benchmarks containing dozens of tasks, suggesting the problem is structural rather than data-related. The implications extend beyond academic integrity. Practitioners selecting models for production rely on SOTA claims to guide adoption decisions. If claims overstate true performance margins or stability, resource allocation follows flawed signals. The research community reinforces the problem through publication norms that reward incremental leaderboard gains with visibility and prestige. The proposed solution requires no additional experiments—merely honest language matching evidence strength. Instead of declaring SOTA based on marginal mean improvements, researchers should report effect sizes, task-by-task consistency, and robustness metrics. This transparency would enable more reliable model selection and accelerate genuine progress by distinguishing real breakthroughs from statistical artifacts.

• →Over 50% of top-model comparisons on major benchmarks lack evidence supporting implicit SOTA superiority claims
• →Marginal improvements in aggregate scores often reflect outlier datasets rather than consistent model superiority
• →SOTA claims routinely lack meaningful effect sizes, task consistency, or robustness to dataset removal
• →The gap between claim strength and evidence requires no new experiments—only more honest reporting of results
• →Current benchmarking practices create incentives for overstated claims that mislead practitioners selecting models