PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation

Researchers introduced PDEAgent-Bench, the first comprehensive benchmark for evaluating AI systems that generate numerical solvers from partial differential equations (PDEs). The benchmark contains 645 test cases across multiple PDE families and finite-element libraries, revealing that while current LLMs can produce runnable code, they substantially fail when accuracy and efficiency requirements are enforced.

PDEAgent-Bench addresses a critical gap in AI evaluation infrastructure by establishing rigorous standards for PDE-to-solver code generation—a domain requiring both mathematical sophistication and practical implementation excellence. The benchmark's staged evaluation framework reflects real-world constraints: generated solvers must not only execute without errors but also meet numerical accuracy tolerances and computational efficiency targets. This approach fundamentally differs from general-purpose code benchmarks that prioritize syntactic correctness over domain-specific reliability.

The research reveals a significant competency gap in current AI agents. While models demonstrate reasonable capability at producing syntactically valid code, their performance drops dramatically when subjected to numerical accuracy verification and runtime constraints. This pattern suggests that language models, despite their broad capabilities, struggle with the interplay between mathematical reasoning, algorithmic selection, and efficient implementation—challenges specific to scientific computing.

For the AI and scientific computing communities, PDEAgent-Bench establishes a reproducible evaluation standard comparable to what benchmarks like MMLU did for general knowledge. Researchers developing AI agents for scientific applications now have quantifiable metrics reflecting practical requirements rather than academic correctness. The multi-library design spanning DOLFINx, Firedrake, and deal.II prevents solution overfitting to single frameworks.

The benchmark's implications extend beyond pure research. As AI systems increasingly assist in scientific and engineering workflows, validated testing methodologies become essential for ensuring reliability in high-stakes applications. PDEAgent-Bench provides infrastructure for advancing agent capabilities toward production-quality scientific code generation, establishing expectations that future systems must clear accuracy and efficiency hurdles alongside basic functionality.

• →PDEAgent-Bench is the first benchmark specifically designed to evaluate AI systems generating PDE numerical solvers across multiple libraries and mathematical categories.
• →Current LLMs produce runnable code frequently but fail substantially when numerical accuracy and computational efficiency requirements are enforced.
• →The staged evaluation framework (executability → accuracy → efficiency) reflects practical constraints absent from general-purpose code benchmarks.
• →The benchmark spans 645 instances across 11 PDE families and 3 major finite-element libraries, preventing solution overfitting to single frameworks.
• →Results indicate significant remaining gaps in AI agent capabilities for scientific computing despite advances in general code generation.