Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas

Researchers evaluated metacognitive monitoring across 33 frontier LLMs using 47,151 MMLU benchmark items, finding significant domain-level variation masked by aggregate performance scores. Applied/Professional knowledge domains showed consistently strong self-monitoring (AUROC .742), while Formal Reasoning and Natural Science proved most challenging, with implications for targeted model deployment.

This comprehensive empirical study addresses a critical blind spot in LLM evaluation: aggregate benchmark scores obscure meaningful performance variations across knowledge domains. By testing 33 models from eight families on 1,500 MMLU items grouped into six domains, researchers quantified how accurately models assess their own confidence across different problem types. The finding that Applied/Professional knowledge achieves reliable high monitoring (top-2 ranking in 64% of models) while Formal Reasoning and Natural Science consistently rank bottom-2 (82% of models) reveals systematic patterns in model self-awareness.

This research builds on growing recognition that LLM capabilities are unevenly distributed across domains. Prior work established that models perform differently on knowledge categories, but this study advances understanding by measuring confidence calibration—whether models know what they know—rather than raw accuracy alone. The coherence analysis confirming the six-domain taxonomy validates the benchmark's pragmatic value, even as a non-latent construct.

For deployment decisions, the domain-level variation has direct consequences. Applications requiring reasoning in formal systems or natural sciences cannot rely on aggregate AUROC metrics to assess model suitability. The observation that within-family clustering varies significantly (Anthropic, Google-Gemini, Qwen show strong coherence; DeepSeek and OpenAI do not) suggests architectural or training differences affect domain-specific self-monitoring patterns. The discovery that invalid binary probes yield normal profiles under verbalized confidence indicates evaluation methodology substantially influences apparent metacognitive performance, requiring careful attention to prompt formatting in production systems.

• →Applied/Professional knowledge domains show reliably strong metacognitive monitoring across 33 LLMs, while Formal Reasoning and Natural Science consistently rank lowest
• →Aggregate benchmark scores mask critical domain-level variation that determines suitability for specialized applications
• →Within-family profile clustering is significant for Anthropic, Google-Gemini, and Qwen but not DeepSeek or OpenAI, indicating architectural differences affect monitoring patterns
• →Evaluation format substantially influences apparent metacognitive performance, as invalid binary probes produced different profiles than verbalized confidence methods
• →Domain screening at benchmark stage is recommended before deployment in specific application areas to ensure appropriate model selection