Power Systems Agent Benchmark: Executable Evaluation of AI Agents in Electric Power Engineering

Researchers introduce the Power Systems Agent Benchmark, an executable evaluation framework for AI agents in electric power engineering with 41 task families across eight engineering domains. The benchmark uses deterministic evaluation to assess whether AI agents can perform real power-system engineering tasks correctly, marking the first major standardized assessment tool for this emerging application area.

The Power Systems Agent Benchmark represents a significant maturation milestone for AI applications in critical infrastructure domains. Unlike traditional language model evaluations that rely on prose grading, this framework employs executable evaluation—checking whether an agent's actions produce correct engineering outcomes rather than assessing the quality of explanations. This shift from text-based assessment to consequence-based evaluation is crucial for infrastructure applications where incorrect answers pose real operational and safety risks.

The benchmark's breadth across eight power engineering domains—spanning power flow, protection systems, stability analysis, microgrids, reliability, power quality, and forecasting—demonstrates the diversity of tasks AI agents must handle in practical engineering environments. Each task grounds itself in documented standards and citable sources, ensuring the benchmark reflects real-world engineering practices rather than synthetic academic problems. The use of private, on-demand synthesized test cases resists data contamination while maintaining inspectable construction processes.

The reference evaluation revealed meaningful performance differentiation among tested agents, with the strongest approaching tier ceilings while smaller open models demonstrated clear capability gaps. Critically, the benchmark's quality control process identified latent evaluator bugs, proving that rigorous benchmarking itself improves system reliability. The architecture allowing evaluator internals to upgrade to simulator-backed checks without modifying task interfaces provides valuable forward compatibility.

For the AI-for-infrastructure sector, this benchmark establishes methodological rigor for assessing agents in safety-critical domains. The framework's success may accelerate adoption of AI agents in power systems while establishing evaluation standards that other critical infrastructure domains—water, transportation, telecommunications—will likely adopt. Organizations developing power-system AI tools now have a standardized way to demonstrate capability and improvement.

• →First executable evaluation benchmark for AI agents in electric power engineering with 41 task families across eight domains
• →Deterministic evaluation checks real engineering consequences rather than grading prose, essential for infrastructure applications
• →Reference evaluation exposed evaluator bugs and showed performance differentiation between command-line agents and open models
• →Private synthesized test cases resist contamination while maintaining inspectable construction for reproducibility
• →Upgradeable evaluator architecture allows integration of simulator-backed checks without changing task definitions