Business Utility of Large Language Models as Exploratory Data Analysis Agents

Researchers evaluated Large Language Models as exploratory data analysis agents in business settings, finding that most configurations lack sufficient repeatability for autonomous deployment despite acceptable average performance. GPT-5.4 with extra-high reasoning emerged as the most reliable option, but the study introduces a 'Business utility' metric combining quality and consistency to assess operational trustworthiness rather than relying solely on average accuracy scores.

This research addresses a critical gap between theoretical LLM capabilities and practical business deployment requirements. While LLMs demonstrate impressive performance on average metrics, organizations deploying these systems for high-stakes analytical tasks—such as supply chain quality analysis—require both accuracy and consistency. The study reveals a sobering reality: many commercially viable models fail repeatability tests despite showing strong mean scores, suggesting that current benchmarking practices inadequately capture real-world risk.

The findings emerge from a methodologically rigorous evaluation using a supply chain simulation where models must identify quality failures from indirect operational signals. This mirrors actual business scenarios where data is noisy, relationships are implicit, and false conclusions carry financial consequences. The introduction of the Business utility metric represents a meaningful contribution to AI evaluation frameworks, acknowledging that variance and condition sensitivity directly impact organizational risk tolerance.

For enterprises considering LLM-powered analytics, these results emphasize the distinction between research-grade performance and production-grade reliability. A model achieving 87% accuracy is fundamentally different from one that achieves 87% accuracy consistently across different data presentations and prompt variations. Organizations will need to invest in extensive testing protocols before adopting LLM agents for autonomous decision-making in supply chain, financial, or operational contexts.

The dominance of GPT-5.4 with enhanced reasoning suggests that model scale and reasoning depth correlate with reliability, but the substantial performance gap between top performers indicates maturation remains incomplete. Future development should prioritize consistency metrics alongside accuracy, particularly for applications where decision volatility creates operational costs.

• →Most LLM configurations fail repeatability tests for business EDA despite acceptable average performance scores.
• →The newly proposed Business utility metric combines quality and consistency into a single risk-adjusted operational measure for trustworthiness assessment.
• →GPT-5.4 with extra-high reasoning achieved the strongest overall profile with 0.6952 business utility, significantly outperforming competitors.
• →Condition sensitivity—how model outputs vary with data representation and prompt clarity—represents an underestimated dimension of production readiness.
• →Organizations should implement comprehensive consistency testing protocols before deploying LLMs for autonomous analytical decision-making.