Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting

Researchers identify a critical failure mode in test-time reinforcement learning (TTRL) where majority voting locks onto incorrect answers, permanently suppressing correct signals in low-ability problems. They introduce TTRL-Guard, a framework using flip-rate monitoring and selective updating to prevent this 'Correct-Answer Extinction Window,' achieving 54% relative improvement on AIME 2025 benchmarks.

This research addresses a fundamental limitation in how modern language models learn during inference. While TTRL using majority voting has shown impressive accuracy gains on mathematical reasoning tasks, the authors demonstrate these improvements mask a concerning pattern: the technique corrupts more problems than it solves by locking onto incorrect answers prematurely. The Correct-Answer Extinction Window represents a critical phase where correct solutions exist in the model's ensemble but get permanently suppressed by majority vote convergence on wrong answers.

The problem stems from how TTRL uses majority voting as a pseudo-label signal. In problems where the model initially generates more incorrect than correct solutions, majority voting quickly solidifies around the wrong answer. Once locked in, this incorrect signal becomes self-reinforcing, irreversibly damaging the model's ability to recover the correct answer. The authors identify Flip Rate—the proportion of changing answers—as a leading indicator of this vulnerability, enabling proactive detection.

TTRL-Guard's three-pronged approach targets this window by scaling rewards based on flip rates, preserving minority correct signals through sampling strategies, and selectively suspending updates on polarized problems. These mechanisms shift the optimization landscape away from majority-vote-induced convergence toward more robust learning. The 54% improvement on AIME 2025 demonstrates meaningful gains beyond surface-level accuracy inflation.

For AI practitioners, this reveals that ensemble-based inference methods require careful monitoring and adaptive mechanisms to prevent pathological convergence patterns. The framework's lightweight design makes it practically implementable across different model architectures.

• →TTRL's majority voting can permanently lock onto incorrect answers, corrupting more problems than it solves in low-ability cases
• →Flip Rate serves as a reliable early warning indicator of the Correct-Answer Extinction Window phenomenon
• →TTRL-Guard achieves 54% relative improvement on AIME 2025 through flip-rate-aware reward scaling and minority-preserving sampling
• →Most reported TTRL accuracy gains reflect sharpening of already-solvable problems rather than genuine learning
• →Adaptive update suspension on polarized problems prevents irreversible damage from incorrect convergence