FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs

Researchers introduce FEM-Bench, a scientific reasoning benchmark designed to evaluate large language models' ability to generate correct finite element method (FEM) code for computational mechanics problems. Despite the simplicity of introductory-level tasks, current state-of-the-art LLMs show inconsistent performance, with Gemini 3 Pro completing 30/33 tasks at least once and GPT-5 achieving 73.8% success on unit test writing.

FEM-Bench addresses a significant gap in AI evaluation frameworks by establishing rigorous metrics for assessing whether LLMs can generate scientifically valid code for physical modeling. Computational mechanics provides an ideal testing ground because it requires strict mathematical adherence, enforces clear physical constraints, and enables objective verification—eliminating subjective evaluation common in other AI benchmarks. The benchmark's design reflects a broader industry shift toward evaluating AI systems on domain-specific technical competencies rather than general reasoning tasks.

The performance data reveals a concerning pattern: even simplified graduate-level computational mechanics problems expose meaningful weaknesses in current leading models. The gap between best-case performance (30/33 one-time completions) and reliable performance (26/33 all five attempts) demonstrates that LLMs generating scientific code remain unreliable, introducing significant risk for researchers and engineers who might deploy these models for actual physical simulations. This unreliability stems from LLMs' difficulty maintaining precise mathematical reasoning and numerical stability across varied problem instances.

For the AI development community, FEM-Bench provides essential infrastructure for tracking progress in scientific reasoning capabilities. As models evolve toward more sophisticated physical reasoning and world modeling—acknowledged goals in LLM advancement—structured evaluation frameworks become critical. This benchmark establishes measurable baselines for comparing future model iterations.

The practical implications extend to developers building AI-assisted scientific computing tools. Until LLMs reliably solve FEM-Bench tasks consistently, automated scientific code generation remains a supplementary tool requiring expert human validation rather than a reliable primary method. Future iterations incorporating more complex problems will likely expose even greater performance gaps.

• →FEM-Bench establishes the first rigorous benchmark for evaluating LLM-generated finite element method code with objective success criteria.
• →State-of-the-art models show significant reliability gaps: Gemini 3 Pro solved 79% of tasks once but only 79% consistently across five attempts.
• →Current LLMs remain unsuitable for autonomous scientific code generation in computational mechanics without expert validation.
• →The benchmark creates measurable baselines for tracking AI progress in physical reasoning and mathematical modeling capabilities.
• →Performance variation across models indicates that scientific reasoning capability is not yet a solved problem in LLM development.