Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning

Researchers propose EGPO, a new framework that improves large reasoning models by incorporating uncertainty awareness into reinforcement learning training. The approach addresses the "uncertainty-reward mismatch" where current training methods treat high and low-confidence solutions equally, preventing models from developing better reasoning capabilities.

• →Current reinforcement learning training for reasoning models ignores intrinsic uncertainty, treating all correct answers equally regardless of confidence levels.
• →EGPO framework integrates uncertainty estimation into training using token-level likelihood entropy as a zero-overhead proxy.
• →The approach preserves correct reasoning while regulating overconfident failures through asymmetric calibration mechanisms.
• →Extensive experiments show substantial improvements in reasoning performance across multiple benchmarks.
• →The framework enables models to better distinguish between what they know and don't know, improving reasoning quality over mere answer memorization.