The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost

Researchers analyzing LLM-based automated scoring found that strategic model selection and reasoning configurations outperform ensemble methods for accuracy. Temperature sampling improved performance, but larger ensemble sizes showed diminishing returns, while higher reasoning effort correlated with better accuracy at varying cost-benefit ratios across model families.

This research addresses a critical operational challenge in deploying LLMs for real-world assessment tasks: how to optimize accuracy while managing computational costs. The findings challenge conventional wisdom that larger ensembles universally improve LLM reliability. By testing 900 student mathematics conversations against human benchmarks, the study demonstrates that thoughtful configuration choices matter more than brute-force ensemble scaling.

The research reflects growing maturity in the LLM evaluation space. As organizations move beyond proof-of-concept deployments, they face tradeoffs between frontier models with superior reasoning capabilities and lightweight alternatives offering better economics. The efficiency frontier analysis revealing that Gemini 3.1 Pro Preview dominates accuracy while GPT-4 Nano/Mini offer superior cost-performance illustrates this tension concretely.

For practitioners developing automated grading systems, content moderation platforms, or other assessment workflows, this work provides empirical guidance on configuration selection. The linear relationship between reasoning effort and accuracy suggests diminishing returns at higher settings, enabling informed resource allocation. Temperature sampling's effectiveness over deterministic inference is particularly relevant for tasks requiring nuanced judgment.

The practical implications extend beyond education technology. Any organization deploying LLMs for scoring, ranking, or classification decisions can apply these findings to model selection and prompt engineering strategies. As model proliferation continues, research distinguishing performance characteristics across cost tiers becomes increasingly valuable for rational deployment decisions.

• →Temperature sampling improved LLM scoring accuracy significantly, but ensemble size increases beyond 3-4 samples showed no meaningful gains.
• →Higher reasoning effort correlated with better accuracy but with different cost-benefit profiles depending on model family.
• →Gemini 3.1 Pro Preview achieved highest accuracy while GPT-4 Nano/Mini provided the best cost-performance ratio for automated scoring tasks.
• →Strategic model and configuration selection outperforms ensemble methods for optimizing LLM-based assessment accuracy.
• →Frontier models justified by superior reasoning capabilities for complex assessment scenarios while lightweight models excel in cost-constrained deployments.