Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning

Researchers introduce a strategy-level evaluation framework for large language models on mathematical reasoning tasks, revealing a significant gap between high answer accuracy and actual reasoning flexibility. While frontier models achieve 95-100% accuracy on single-solution prompts, they recover substantially fewer problem-solving strategies than human references when asked to generate multiple approaches, with only 39-71% coverage depending on the model and iteration count.

Current benchmarking of large language models relies heavily on final-answer accuracy, a metric that masks critical limitations in reasoning flexibility. This research exposes a fundamental decoupling: models can produce correct answers without demonstrating the diverse problem-solving strategies that human mathematicians employ. The study analyzed four frontier models—Gemini, DeepSeek, GPT, and Claude—against 217 strategy families derived from Art of Problem Solving resources, finding that strategy generation varies dramatically across models, from 110 distinct valid strategies for Claude to 184 for Gemini. The largest deficiencies appear in geometry and number theory domains, suggesting domain-specific reasoning limitations.

This evaluation framework addresses a critical gap in AI assessment methodology. Traditional accuracy-focused metrics reward pattern matching and memorization over genuine mathematical reasoning, potentially overstating model capabilities. The research demonstrates that models generate some novel valid strategies absent from human references, indicating creative problem-solving potential coexists with significant strategy blindspots. Robustness testing across multiple runs reveals diminishing returns, with the strongest model recovering only 71% of reference strategies after three attempts, suggesting fundamental limitations rather than sampling variability.

For AI developers and researchers, this work highlights that improving accuracy alone provides incomplete progress toward robust reasoning systems. The framework enables more granular evaluation of reasoning quality and identifies systematic weaknesses in specific mathematical domains. The findings suggest that current models may struggle with complex, real-world problems requiring multiple approaches or when initial strategies fail. Future LLM development should prioritize strategy diversity and reasoning flexibility alongside accuracy optimization.

• →Frontier LLMs achieve 95-100% answer accuracy but recover only 39-71% of human reference strategies, revealing a critical accuracy-flexibility decoupling.
• →Gemini generates the most distinct strategies (184), while Claude generates the fewest (110), indicating substantial performance variation across model architectures.
• →Geometry and number theory show the largest strategy coverage gaps, identifying domain-specific reasoning weaknesses in current models.
• →Models generate 50 novel valid strategies absent from human references, demonstrating creative problem-solving capacity alongside incomplete human strategy coverage.
• →Repeated runs show diminishing returns in strategy discovery, suggesting fundamental model limitations rather than sampling or prompt variation issues.