MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries

Researchers introduced MathlibPR, a benchmark dataset derived from real Mathlib4 pull request histories, to evaluate whether large language models can assist in reviewing mathematical code contributions. Testing revealed that current LLMs struggle to distinguish merge-ready pull requests from those that passed builds but were revised or rejected, highlighting limitations in automated code review for formal mathematics.

The Lean/Mathlib ecosystem represents a critical infrastructure layer for formal mathematics and LLM-assisted reasoning, yet its growth faces a human bottleneck: the code review process. MathlibPR directly addresses this constraint by creating a structured benchmark from actual PR data, enabling systematic evaluation of whether LLMs can reduce reviewer burden. This matters because Mathlib maintainers currently spend significant time on subjective quality assessments, and automating even partial aspects could accelerate development velocity.

The research reveals a gap between surface-level code quality metrics and merge-readiness. LLMs can often generate syntactically correct Lean code, but determining whether contributions align with Mathlib conventions, architectural philosophy, and long-term maintainability requires deeper understanding. The benchmark's staged evaluation protocol—distinguishing between build-passing and merge-ready PRs—captures this nuance that simpler metrics miss. This finding applies beyond Mathlib to any large codebases with implicit quality standards.

For the formal verification ecosystem, this research signals both opportunity and challenge. While current models underperform, the benchmark itself becomes a training signal for future systems. Organizations building mathematical AI tools, formal verification platforms, and code-generation systems should recognize that passing compilation tests differs fundamentally from production readiness. The work suggests reward models trained on historical PR acceptance data could eventually enable more effective LLM steering, but substantial progress remains before automated reviewers match human judgment on nuanced architectural decisions.

• →MathlibPR provides the first standardized benchmark for evaluating LLM performance on formal mathematics code review tasks
• →Current LLMs including DeepSeek, Qwen, and Claude struggle to distinguish merge-ready contributions from build-passing but ultimately rejected pull requests
• →The gap between syntactic correctness and architectural soundness represents a core limitation in applying LLMs to formal mathematics infrastructure
• →Historical PR data can serve as supervised training signals for developing better reward models and reviewer-assistant systems
• →Addressing Mathlib's review bottleneck through LLM assistance remains an unsolved challenge requiring deeper model understanding of community standards