Benchmarks in Leipzig

Researchers at the Max Planck Institute compiled 100 research-level mathematics questions to benchmark large language models' reasoning capabilities. Through three evaluation stages, only 2 questions remained unsolved by advanced LLMs, indicating significant progress in AI mathematical reasoning.

The Benchmarks in Leipzig project represents a systematic effort to measure the mathematical reasoning capabilities of state-of-the-art language models through rigorous empirical testing. The collaborative workshop brought together 49 mathematicians to create a curated dataset of genuinely difficult problems, establishing a reliable evaluation framework for AI systems that goes beyond standard benchmarks. This methodological rigor matters because previous AI evaluations often relied on existing datasets that models may have encountered during training, potentially inflating performance metrics.

The dramatic reduction in unsolved questions across three stages—from 41 to 16 to just 2—reflects the rapid improvement curve in LLM reasoning capabilities over the past year. The progression from single attempts to multiple runs to specialized reasoning models demonstrates that success depends partly on computational resources and inference strategies rather than model architecture alone. This finding suggests that current LLMs possess latent mathematical reasoning abilities that manifest more clearly under optimal evaluation conditions.

For the broader AI industry, this research provides concrete evidence supporting claims about LLM advancement in abstract reasoning tasks. The results validate investments in reasoning-focused model variants and inference optimization techniques. However, the persistence of 2 unsolved problems indicates fundamental limitations remain, suggesting researchers should focus on understanding failure modes rather than declaring general reasoning solved.

Looking forward, follow-up benchmarks will likely test these models on newer, harder problems and explore whether improvements generalize to other domains like formal theorem proving and scientific discovery, which remain commercially valuable applications.

• →Only 2 of 100 research-level math questions remained unsolved after three evaluation stages with advanced LLMs.
• →Multiple-run evaluations with the same models solved significantly more problems than single attempts, suggesting solution strategies matter.
• →The benchmark provides a rigorous framework for measuring mathematical reasoning that accounts for training data contamination issues.
• →Results indicate LLMs possess mathematical abilities that improve with computational resources and specialized reasoning approaches.
• →Two persistent failures suggest fundamental limitations still exist despite impressive overall performance improvements.