Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation

Researchers adapted clinical psychology's Reliable Change Index to evaluate LLM performance across model versions, revealing that aggregate accuracy gains mask substantial item-level volatility. Testing Llama 3→3.1 and Qwen 2.5→3 showed bidirectional changes with large effect sizes, where improvements in low-accuracy domains offset deteriorations in high-accuracy ones, suggesting current evaluation methods underestimate model instability.

This research introduces a methodological framework that fundamentally challenges how the AI community evaluates large language model improvements. By applying the Reliable Change Index—a statistical tool from clinical psychology measuring whether individual outcomes change beyond measurement error—the authors expose a critical gap in LLM evaluation practices. While both model upgrades showed modest aggregate gains (1.6-2.8 points), item-level analysis revealed dramatic volatility: roughly one-third of items improved while one-quarter to one-third deteriorated, with median effect sizes reaching 0.50-0.90 in probability shifts.

The findings highlight a structural problem in benchmark reporting. When vendors announce accuracy improvements, these figures represent net outcomes of offsetting gains and losses across different domains and difficulty levels. For Llama, physics performance collapsed while other domains improved. Qwen exhibited the inverse pattern with law domains. This asymmetric churn by difficulty—where easy items stay stable while hard items randomly improve or degrade—suggests model changes don't represent genuine capability gains but rather redistributed performance across the problem space.

The practical implications are substantial for model selection and deployment. Single-shot greedy evaluation, the current industry standard, misses 42% of meaningful changes and falsely identifies 25% of stable items as changed. Organizations adopting new model versions cannot rely on aggregate benchmarks to predict domain-specific behavior. This research establishes a framework for more rigorous evaluation but exposes uncomfortable truths: published accuracy improvements lack granularity and reliability assessment. Going forward, the field must adopt churn rate reporting alongside aggregate metrics to provide stakeholders honest assessments of model stability rather than potentially misleading aggregate gains.

• →Aggregate accuracy improvements mask substantial bidirectional item-level changes, with 28-39% of items deteriorating in each model update.
• →Current single-shot evaluation methodology misses 42% of reliably changed items and incorrectly flags 25% of unchanged items as improved.
• →Model upgrades show asymmetric performance shifts by difficulty level, with low-accuracy domains improving while high-accuracy domains degrade.
• →Family-specific domain reversals emerge between model versions: Llama lost physics capability while Qwen lost law domain performance.
• →Researchers recommend reporting churn rate metrics alongside aggregate accuracy to provide transparent assessment of model stability and real-world reliability.