Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis

Researchers have developed a new automated pipeline that generates challenging math problems by first identifying specific mathematical concepts where LLMs struggle, then creating targeted problems to test these weaknesses. The method successfully reduced a leading LLM's accuracy from 77% to 45%, demonstrating its effectiveness at creating more rigorous benchmarks.

• →New AI-driven pipeline automatically generates difficult math problems by analyzing LLM weaknesses rather than relying on manual benchmark creation.
• →The method uses AI-generated hypotheses to identify specific math concepts where LLMs are most error-prone.
• →Generated problems reduced Llama-3.3-70B-Instruct's accuracy to 45% compared to 77% on existing MATH benchmark.
• →The pipeline is adaptable beyond mathematics and can be applied to test LLM capabilities across various domains.
• →Higher hypothesis accuracy correlates with increased problem difficulty, validating the approach's effectiveness.