Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not Causal

Researchers discovered that large language models internally detect their own reasoning errors with 95% accuracy but verbally express unwarranted confidence in flawed outputs. Despite this hidden awareness, four intervention strategies failed to correct the errors, indicating the signal reflects computation quality rather than a mechanism that can be leveraged for improvement.

This research exposes a fundamental disconnect between what large language models know internally and what they communicate externally. While chain-of-thought prompting relies on the assumption that a model's reasoning accurately reflects its computational process, these findings demonstrate that models contain sophisticated error-detection mechanisms operating in hidden layers—detectable through linear probes with near-perfect accuracy—yet mask this awareness through confident verbal outputs. This paradox holds across multiple model families and scales, from 1.5B to 72B parameter models, including advanced reasoning systems like DeepSeek-R1.

The inability to weaponize this hidden knowledge represents a critical boundary in mechanistic interpretability research. The authors tested four distinct intervention approaches—activation steering, probe-guided selection, self-correction, and activation patching—yet none successfully converted diagnostic signals into corrective action. Notably, activation patching caused output coherence to collapse, suggesting that error representations are deeply intertwined with generative capacity rather than separable components. This contrasts sharply with prior successes in editing factual knowledge, revealing that reasoning errors occupy a fundamentally different representational space than semantic facts.

These findings carry implications for AI safety and reliability. If models can detect errors internally but cannot access or utilize that knowledge, the path toward more trustworthy AI systems becomes more constrained. Engineers cannot rely on simply extracting hidden signals; instead, fundamental changes to model architecture or training methodology may be necessary. The work establishes that interpretability breakthroughs in one domain—factual knowledge—do not automatically transfer to others like reasoning correctness.

• →LLMs detect their own reasoning errors with 95% accuracy in hidden states but express 90%+ confidence in wrong outputs externally.
• →The error-detection signal is diagnostic only—four intervention strategies failed to convert it into corrective action.
• →Hidden error awareness appears across model families (Qwen, Llama, Phi, DeepSeek-R1) and scales from 1.5B to 72B parameters.
• →Error representations differ fundamentally from factual knowledge representations, limiting mechanistic interpretability advances.
• →Attempts to steer or patch error signals destroy output coherence, suggesting reasoning errors are structurally integral to generation.