ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?

Researchers introduced ORAgentBench, a benchmark testing whether AI agents can autonomously solve complex operations research tasks end-to-end. Testing 14 frontier agent-model configurations revealed significant limitations: the best agent solved only 35.51% of tasks and 20.59% of hard tasks, with failures stemming from missed operational rules, weak solution construction, and insufficient optimization—indicating AI agents remain far from production-ready OR work.

The release of ORAgentBench represents a critical evaluation framework that exposes a substantial capability gap in autonomous AI agents tasked with operations research. Unlike previous benchmarks that isolate modeling from solving or rely on simplified instances, this evaluation tests the full workflow from raw operational data through validated decision-making. The benchmark's rigor—including hidden validators, hard-constraint checking, and objective quality metrics—provides a realistic assessment of what current agents can actually accomplish in real-world scenarios.

This research emerges amid growing enthusiasm for deploying LLMs as autonomous agents across enterprise domains. Operations research, which optimizes complex resource allocation and scheduling problems, represents a high-stakes application area where suboptimal solutions directly impact business outcomes. Previous benchmarks failed to capture the practical challenges agents face: interpreting multi-file datasets, understanding implicit operational constraints, constructing feasible initial solutions, and iteratively improving solutions toward quality thresholds.

The findings expose strategic weaknesses that go beyond simple coding errors. Agents frequently miss operational rules embedded in natural language briefs, formulate problems in brittle ways that fail under real-world constraints, and lack systematic approaches to solution construction and improvement. Even when procedural OR training improves feasibility rates on hard tasks, it fails to translate into better overall success rates or solution quality, suggesting agents struggle with higher-level strategic reasoning.

For organizations considering AI-driven optimization systems, the results indicate current frontier models remain unsuitable for autonomous end-to-end OR work without substantial human oversight. The research suggests future progress requires moving beyond generating plausible code toward building agents that reliably produce operationally sound, high-quality decisions—a shift requiring deeper integration of domain knowledge and multi-step verification mechanisms.

• →Best-performing agents solve only 35.51% of ORAgentBench tasks, with hard task success dropping to 20.59%, indicating current LLM agents lack production-ready capabilities for operations research.
• →Failure analysis identifies strategic weaknesses beyond coding errors: missed operational rules, brittle problem formulations, weak solution construction, and insufficient optimization techniques.
• →OR-specific procedural training improves hard-task feasibility but fails to reliably enhance solution quality or overall pass rates, suggesting agents lack higher-level strategic reasoning.
• →The benchmark's execution-grounded approach—testing full workflows with hidden validators and multi-file datasets—reveals limitations that simplified benchmarks overlook.
• →Organizations considering autonomous AI optimization systems should expect substantial human oversight remains necessary for mission-critical operations research tasks.