BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

BenchEvolver is an AI framework that automatically generates harder variants of existing coding problems to address benchmark saturation, where frontier LLMs now achieve 99% accuracy on standard tests. By evolving solutions rather than creating problems from scratch, it produces verifiable, diverse tasks that maintain challenge even for their generating models, enabling both better evaluation and improved training signals.

Benchmark saturation represents a critical bottleneck in AI development. When frontier models solve 99% of existing coding benchmarks, researchers lose the ability to measure meaningful performance differences or extract training signals. BenchEvolver addresses this by inverting the typical benchmark construction process—instead of writing problems first and solutions second, it starts with correct solutions and systematically transforms them into harder variants, then derives new problem statements and test cases. This approach grounds generation in executable semantics, ensuring correctness by construction.

The framework's effectiveness is demonstrated through LiveCodeBench-Plus, a curated benchmark where frontier model performance ranges from 27.5% to 62.6%, restoring meaningful discrimination among strong models. Critically, the evolved tasks remain challenging for the models that generate them, preventing gaming through self-referential optimization. The reinforcement learning results show practical utility: training on evolved tasks yields 8.7 percentage point improvements on held-out benchmarks, substantially outperforming seed-only baselines.

This work impacts the AI development cycle fundamentally. As models improve, static benchmarks become artifacts of the past; dynamic, solution-centric benchmark generation may become standard infrastructure. For developers and researchers, this reduces the human effort required to construct challenging evaluation datasets. For model developers, evolved benchmarks provide both honest evaluation capabilities and higher-quality training data. The approach is generalizable beyond coding—any domain with verifiable solutions could benefit from similar evolution strategies.

• →BenchEvolver automatically generates harder coding tasks by evolving existing solutions rather than creating problems from scratch, addressing benchmark saturation where frontier models achieve 99% accuracy.
• →The framework ensures correctness by deriving problem statements from evolved solutions, enabling scalable construction of difficult, valid, and diverse tasks.
• →LiveCodeBench-Plus curates 91 problems where frontier models achieve 27.5%-62.6% Pass@1, restoring clear differentiation among state-of-the-art coding models.
• →Reinforcement learning on evolved tasks improves held-out performance by up to 8.7 percentage points, demonstrating practical training signal value.
• →Evolved tasks remain challenging for generating models, enabling self-improvement without gaming or circular dependencies.