Expected Value Alignment for Generative Reward Modeling in Formal Mathematics Verification

Researchers introduce Expected Value Alignment (EVA), a novel reward-modeling technique that enables Large Language Models to provide continuous numerical scores while maintaining human-readable text output for formal mathematics verification in Lean 4. The method bridges a critical gap between discrete generative outputs and continuous value assessment needed for reinforcement learning in theorem proving systems.

The convergence of large language models and formal verification systems represents a significant frontier in AI development, where mathematical correctness becomes paramount. Traditional approaches to reward modeling present a fundamental tension: value-head architectures deliver continuous scores but obscure the reasoning process, while generative models preserve interpretability but struggle with numeric precision when splitting values across multiple tokens. EVA resolves this trade-off through an elegant mathematical approach, extracting continuous expectations from token-level probability distributions while preserving the discrete, interpretable structure of the output.

This advancement addresses a bottleneck in scaling AI-assisted theorem proving systems. As reinforcement learning and search-based methods increasingly train on intermediate reasoning steps, the quality of process reward models directly determines system performance. Prior approaches suffered from discretization artifacts—quantization errors introduced when converting floating-point values to token sequences. EVA's logit-based scoring mechanism inherently captures the model's uncertainty and confidence levels without artificial quantization, improving alignment between the reward signal and actual reasoning quality.

The practical implementation in Leibniz demonstrates immediate applications for Lean 4 formal verification, where mathematical proofs require step-by-step validation. The dual-objective training combining language modeling with mean squared error loss creates a system that both generates coherent critiques and produces reliable continuous scores. For developers working on AI-assisted mathematics and verification systems, EVA offers a scalable foundation for training more capable reasoning models. The technique's applicability extends beyond formal mathematics to any domain requiring both interpretable outputs and continuous value assessment, potentially influencing how future AI systems balance explainability with quantitative evaluation.

• →EVA enables continuous reward scoring from discrete generative outputs by computing expectations over token logits, eliminating discretization artifacts.
• →The method preserves human-readable JSON-formatted reasoning while supporting the continuous value signals needed for reinforcement learning.
• →Leibniz implementation demonstrates practical effectiveness for Lean 4 theorem proving, outperforming zero-shot and baseline reward models.
• →The dual training objective combining language modeling with MSE loss creates systems that are both interpretable and quantitatively precise.
• →This approach addresses a critical scaling bottleneck for AI systems that require process-level reward evaluation in complex reasoning tasks.