Distributional Process Reward Models: Calibrated Prediction of Future Rewards via Conditional Optimal Transport

Researchers propose using conditional optimal transport to improve calibration of Process Reward Models (PRMs) used in AI inference-time scaling, addressing the problem of overestimated success probabilities. The method enables better confidence bounds for mathematical reasoning tasks and improves downstream performance in Best-of-N selection frameworks.

Process Reward Models have emerged as critical components in scaling inference-time compute for AI systems, particularly in reasoning tasks where multiple solution paths can be explored. However, PRMs suffer from systematic miscalibration, tending to overestimate the probability of success, which degrades their utility in guiding computational resource allocation. This paper addresses a fundamental reliability problem that undermines the practical deployment of these models in high-stakes reasoning applications.

The proposed solution applies conditional optimal transport, a mathematical framework from optimal transport theory, to learn monotonic conditional quantile functions. This approach differs from standard calibration techniques by providing structural guarantees on quantile estimates while remaining computationally efficient. The method conditions calibration on PRM hidden states, enabling instance-adaptive confidence bounds rather than one-size-fits-all adjustments. This sophistication matters because different reasoning problems may exhibit different uncertainty characteristics.

The evaluation demonstrates meaningful improvements across MATH-500 and AIME benchmarks, with particularly strong results when PRMs have reliable ranking signals. Integration into instance-adaptive scaling frameworks shows the method generalizes beyond isolated calibration improvements to downstream task performance gains. However, the results reveal context-dependent effectiveness, suggesting the method works best with certain PRM architectures or training approaches.

This work represents incremental but valuable progress in making inference-time scaling more reliable and efficient. As AI systems increasingly rely on test-time compute for complex reasoning, calibration quality directly impacts both resource efficiency and solution reliability. The contribution bridges mathematical rigor with practical applicability, establishing optimal transport as a principled approach for uncertainty quantification in neural reward models.

• →Conditional optimal transport provides structural guarantees for calibrating Process Reward Models that overestimate success probabilities.
• →The method enables flexible, instance-adaptive confidence bounds without requiring separate calibration models.
• →Evaluation on mathematical reasoning benchmarks shows substantial calibration improvements over standard quantile regression.
• →Integration with instance-adaptive scaling frameworks yields practical downstream performance gains on Best-of-N selection.
• →Effectiveness depends on PRM quality, with best results when models have reliable ranking signals.