Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory

Researchers introduce a new framework to evaluate how well Large Language Models understand their own knowledge limitations, finding that traditional confidence metrics miss key differences between models. The study reveals that models showing similar accuracy can have vastly different metacognitive abilities - their capacity to know what they don't know.

• →Traditional LLM confidence metrics conflate knowledge accuracy with self-awareness of knowledge limitations.
• →Models with similar performance can have dramatically different metacognitive efficiency ratios.
• →Mistral achieved highest accuracy but lowest self-awareness, while other models showed better metacognitive abilities.
• →Metacognitive efficiency varies significantly across different knowledge domains within the same model.
• →The new meta-d' framework provides better insights for AI model selection and human-AI collaboration decisions.