AIBearisharXiv – CS AI · 7h ago7/10
🧠
Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
A new study reveals that standard single-run accuracy metrics for large language models significantly overstate their real-world reliability on programming tasks, with gaps reaching 17.8 percentage points when measuring consistency across repeated invocations. The research introduces a repeated-run evaluation protocol showing that while popular benchmarks emphasize one-time success rates, deployment environments require stable outputs—a critical distinction that current evaluation standards overlook.