The Dunning-Kruger Effect in Large Language Models: An Empirical Study of Confidence Calibration

A new study reveals that large language models exhibit patterns similar to the Dunning-Kruger effect, where poorly performing AI models show severe overconfidence in their abilities. The research tested four major models across 24,000 trials, finding that Kimi K2 displayed the worst calibration with 72.6% overconfidence despite only 23.3% accuracy, while Claude Haiku 4.5 achieved the best performance with proper confidence calibration.

• →Study of four major LLMs reveals significant confidence calibration issues across 24,000 experimental trials.
• →Kimi K2 exhibits severe overconfidence with 72.6% calibration error despite only 23.3% accuracy.
• →Claude Haiku 4.5 demonstrates best performance with 75.4% accuracy and lowest overconfidence at 12.2%.
• →Poorly performing AI models show markedly higher overconfidence, mirroring human Dunning-Kruger cognitive bias.
• →Findings raise important safety concerns for deploying LLMs in high-stakes applications where accuracy matters.