LemmaBench: A Live, Research-Level Benchmark to Evaluate LLM Capabilities in Mathematics

Researchers have developed LemmaBench, a new benchmark for evaluating Large Language Models on research-level mathematics by automatically extracting and rewriting lemmas from arXiv papers. Current state-of-the-art LLMs achieve only 10-15% accuracy on these mathematical theorem proving tasks, revealing a significant gap between AI capabilities and human-level mathematical research.

• →LemmaBench creates an updatable benchmark using real mathematical research from arXiv rather than static contest problems.
• →The system automatically extracts lemmas and rewrites them into self-contained mathematical statements.
• →Current top LLMs achieve only 10-15% accuracy in theorem proving on research-level mathematics.
• →The benchmark can be regularly updated with new problems while preserving previous versions for training.
• →Results show a large gap remains between current AI capabilities and human-level mathematical research abilities.