StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs

Researchers introduced StatABench, a comprehensive benchmark for evaluating LLMs' statistical analysis capabilities across 434 questions and tasks. Evaluations reveal significant performance gaps, with GPT-5.1 achieving only 68.6% accuracy on closed-ended questions and top agent frameworks scoring 61.86% on complex modeling tasks, exposing persistent weaknesses in tool-grounded reasoning and methodological decision-making.

StatABench addresses a critical evaluation gap in LLM development by providing the first large-scale, multi-format benchmark for assessing statistical analysis proficiency. The benchmark's dual-component design—combining 404 structured questions across 18 statistical topics with 30 real-world modeling challenges from professional competitions—reflects the breadth and complexity required for practical statistical work. This comprehensive approach matters because statistical analysis underpins decision-making across finance, healthcare, research, and data science, making reliable LLM performance essential for enterprise adoption.

The research builds on growing recognition that existing LLM evaluations oversimplify complex cognitive tasks. Prior benchmarks focused narrowly on knowledge recall rather than applied reasoning, tool integration, and methodological judgment. StatABench's inclusion of multiple question formats and LLM-as-Judge validation protocols represents methodological rigor that the field needs as LLMs move from prototype to production systems.

The performance results carry significant implications for organizations considering LLM deployment in analytical roles. A 68.6% ceiling for GPT-5.1—the field's leading model—suggests that autonomous statistical analysis remains unreliable, requiring human oversight and verification. The 6-8 percentage point gap between top commercial and open-source models highlights the competitive advantage maintained by larger players, though open-source models' 60%+ performance suggests viable alternatives for resource-constrained teams.

Looking forward, organizations should expect incremental LLM improvements in statistical capability but anticipate continued reliance on human domain experts for critical analyses. The research signals growing investment in specialized evaluation frameworks that measure real-world applicability rather than benchmark gaming, indicating the field's maturation toward production-grade standards.

• →StatABench introduces the first comprehensive benchmark combining 404 closed-ended questions and 30 open-ended modeling tasks to evaluate LLM statistical analysis capabilities.
• →Even GPT-5.1 achieves only 68.6% accuracy on structured statistical questions, indicating significant limitations in current LLM reliability for analytical work.
• →The benchmark reveals persistent gaps in tool-grounded reasoning, methodological decision-making, and end-to-end statistical modeling across all tested LLMs.
• →Open-source models reach 60.6% accuracy, demonstrating viable alternatives though maintaining notable performance gaps compared to commercial leaders.
• →Results suggest LLMs cannot yet function autonomously for critical statistical analysis, requiring continued human expert oversight in production environments.