An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models

Researchers discovered that large reasoning models (LRMs) exhibit a significant production-evaluation gap, scoring as low as 48% when evaluating flawed reasoning despite near-perfect solution generation. Using the VAIR dataset, the study reveals that LRMs suffer from answer confirmation bias—they verify conclusions rather than rigorously evaluate reasoning steps—unlike humans who perform similarly at both tasks.

This research exposes a critical vulnerability in how frontier AI models approach reasoning validation. While LRMs demonstrate remarkable capability in producing lengthy, coherent chains of thought to solve complex problems, they fundamentally fail at the inverse task: determining whether someone else's reasoning is sound. The 48% evaluation score represents a stark contrast to human performance, where people show only 6% performance degradation between solving and grading equivalent problems.

The findings emerge from a carefully constructed experimental framework. The VAIR dataset deliberately isolates reasoning quality from answer correctness—problems contain valid final answers despite trivial logical flaws in the derivation. This design prevents LRMs from relying on answer-checking shortcuts. Through chain-of-thought analysis and linear probes, researchers identified that models engage in answer confirmation bias: they locate the correct answer, then fabricate supporting rationales rather than systematically verify each logical step. Causal patching experiments confirmed that manipulating answer representations directly flips model verdicts, indicating the models' evaluations hinge on answer validity rather than reasoning validity.

For AI development, this reveals a structural limitation in current training paradigms. Reinforcement learning approaches that reward models for producing correct answers inadvertently train confirmation bias into evaluation capabilities. The models learn that correct answers justify any reasoning path, rather than developing robust step-by-step verification protocols.

This limitation matters for deployment scenarios requiring genuine reasoning validation—mathematics verification, code review automation, and safety-critical system audits. The research suggests that improving LRM evaluation capabilities requires fundamentally different training objectives than those optimizing for reasoning production, potentially demanding explicit anti-confirmation-bias mechanisms or alternative verification architectures.

• →Large reasoning models show a 52-percentage-point gap between production (98%) and evaluation (48%) performance on flawed-but-correct reasoning problems.
• →LRMs employ answer confirmation bias, fabricating rationalizations to justify correct answers rather than rigorously evaluating logical steps independently.
• →Linear probes and causal patching reveal that model activations encode answer validity rather than reasoning validity, directly controlling evaluation verdicts.
• →Current LRM training paradigms incentivize answer-correctness optimization without developing robust reasoning verification capabilities.
• →This evaluation deficit poses risks for AI applications requiring genuine logic validation in mathematics, code review, and safety-critical domains.