Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization

Researchers introduce ISPO (Intrinsic Signal Policy Optimization), a new reinforcement learning method that improves long-chain reasoning in large language models by densifying reward signals with intrinsic metrics derived from the model's own probabilities. The approach addresses critical failure modes in existing GRPO-based methods and shows consistent improvements across mathematical reasoning benchmarks.

This research addresses fundamental inefficiencies in current reinforcement learning approaches for training language models on complex reasoning tasks. The paper identifies two concrete failure modes—Zero-Advantage Collapse and Hallucinated Certainty—that plague existing Group Relative Policy Optimization methods relying on binary rewards. By introducing intrinsic signals computed from the policy's conditional probabilities, ISPO creates a denser feedback landscape that guides model training more effectively.

The development emerges from the broader push to improve AI reasoning capabilities through verifiable reward systems. As language models tackle increasingly difficult mathematical and logical problems, existing training methods reveal architectural weaknesses where all model rollouts converge to identical outcomes, eliminating gradient signals necessary for learning. The hallucination problem compounds this by allowing models to express unwarranted confidence in incorrect solutions, a particularly dangerous failure mode in mathematical domains where certainty signals matter.

For the AI research community and companies building reasoning-capable models, ISPO represents a meaningful advancement in training efficiency and quality. The consistent outperformance across multiple benchmarks and base model sizes suggests broader applicability beyond the tested domains. The largest gains on hardest problems indicate the method addresses exactly where current approaches struggle most, making it particularly valuable for frontier model development.

The research trajectory points toward more sophisticated reward design beyond simple binary outcomes. Future work likely explores how these intrinsic signals generalize to other domains requiring step-by-step reasoning, and whether similar densification techniques improve other policy optimization approaches. The methodology's foundation in information-theoretic principles rather than task-specific engineering suggests potential for broader adoption in reasoning-focused AI systems.

• →ISPO solves Zero-Advantage Collapse and Hallucinated Certainty failure modes in current GRPO-based reinforcement learning by densifying rewards with intrinsic probability-based signals
• →The method combines sequence-level informativeness signals with token-level directional rewards to guide model training more effectively on mathematical reasoning tasks
• →Empirical results show consistent improvements across three base models and five benchmarks, with largest gains on hardest problems where existing methods fail most frequently
• →Intrinsic signals are computed entirely from the policy's conditional probabilities, eliminating need for external reward models or task-specific engineering
• →The approach addresses a critical bottleneck in scaling language model reasoning capabilities, relevant for developing more reliable AI systems