EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

Researchers introduce EngiAI, a multi-agent LLM framework with a comprehensive benchmark suite for evaluating AI systems on complex engineering design tasks combining simulation, retrieval, and manufacturing. The framework reveals significant performance gaps between proprietary models (96-97% task completion) and open-source alternatives (55-78%), with conditional reasoning emerging as a critical failure point.

EngiAI addresses a critical gap in AI evaluation methodology by moving beyond single-task benchmarks to assess multi-agent systems handling real-world engineering workflows. This matters because LLM-driven engineering represents a substantial commercial opportunity, yet existing evaluation frameworks fail to capture the complexity of coordinated agent behavior across simulation, knowledge retrieval, and infrastructure orchestration. The benchmark's three-dimensional approach—workflow prompting, retrieval-augmented generation, and HPC job management—provides practitioners with actionable insights into where agent systems fail under realistic conditions.

The performance data carries important implications for AI development trajectories. Proprietary models' near-complete success on structured tasks contrasts sharply with open-source models' struggles, particularly on conditional branching scenarios where task completion drops to 20-53%. This performance cliff suggests that reasoning about conditional logic and maintaining instruction fidelity across long workflows remains fundamentally challenging for smaller models. The RAG benchmark results are especially revealing: retrieval augmentation produces near-perfect scores (1.0) while unaided systems score near zero, demonstrating that engineering applications require hybrid approaches combining retrieval with reasoning.

For stakeholders developing AI infrastructure, EngiAI's findings indicate that engineering automation workflows demand models capable of complex multi-step reasoning and context retention. The HPC orchestration results—one model achieving 100% completion versus another at 50%—suggest that seemingly minor differences in instruction-following capabilities compound catastrophically in production environments. Organizations evaluating LLM backends for engineering applications should prioritize models demonstrating robust conditional reasoning and long-context coherence rather than optimizing purely for general capability metrics.

• →Proprietary LLM models achieve 96-97% task completion on structured engineering workflows while open-source 4B models reach only 55-78%, indicating significant capability gaps
• →Conditional branching logic emerges as the most challenging element across benchmarks, with completion rates dropping to 20-53% on conditional reasoning tasks
• →Retrieval-augmented generation proves essential for engineering applications, with RAG-enabled systems scoring near-perfect (1.0) versus near-zero without retrieval components
• →Multi-step instruction following degrades significantly over long-running workflows, with HPC orchestration performance ranging from 50-100% completion across different models
• →EngiAI's multi-agent supervisor architecture successfully coordinates seven specialized agents for topology optimization, document retrieval, and hardware control tasks