Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation

Researchers identify a critical blind spot in pass@k, the standard metric for evaluating math reasoning difficulty in large language models. Their analysis reveals that 10-23% of problems marked as unsolvable through sampling can actually be solved using deterministic inference with activation grafting perturbations, suggesting current difficulty assessments systematically underestimate model capabilities.

This research exposes a fundamental measurement problem in how the AI community evaluates mathematical reasoning capabilities. Pass@k sampling—repeatedly generating solutions and checking if any reaches the correct answer—has become the de facto standard for assessing problem difficulty and driving critical downstream applications including reinforcement learning, data curation, and model training. However, the study demonstrates this metric has a persistent blind spot precisely where it matters most: the hardest problems.

The researchers tested eight mathematical reasoning datasets (GSM8K and MATH across four open-weight models) and found that problems deemed unsolvable after six sampling attempts were frequently solvable through deterministic methods combining greedy decoding with cheap residual-stream perturbations via activation grafting. The recovery rate—10.3-22.9% of supposed failures—is substantial and scales with computational budget. Critically, greedy decoding alone solves less than 6% of these problems, suggesting the difficulty lies not in fundamental model incompetence but in how inference procedures interact with internal representations.

This discovery has significant implications for the machine learning community. Researchers may be systematically misclassifying problem difficulty, potentially misdirecting optimization efforts toward problems the models partially understand. RL training, synthetic curricula, and verifier development all rely on accurate difficulty signals; misjudging which problems are truly hard versus merely hard-to-sample could waste resources on inefficient training strategies. The mechanistic distinctness across perturbations (Jaccard similarity ≤0.47) indicates models preserve multiple solution pathways internally, suggesting current sampling-based evaluation misses structural capabilities. Organizations using pass@k for model selection or benchmark comparisons may be making decisions based on incomplete information about actual capabilities.

• →Standard pass@k sampling misclassifies 10-23% of hard math problems as unsolvable when they're actually solvable through deterministic inference methods
• →Activation grafting with residual-stream perturbations recovered solutions that greedy decoding alone cannot find, revealing hidden model capabilities
• →The blind spot affects critical downstream applications including RL training, data curation, and curriculum generation that depend on accurate difficulty signals
• →Models maintain structurally identifiable solution pathways in internal representations that sampling-based inference fails to access
• →Recovery scales with computational budget, suggesting the problem is inference methodology rather than fundamental model limitation