ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimization

Researchers introduced ConstraintBench, a new benchmark testing whether large language models can directly solve constrained optimization problems without external solvers. The study found that even the best frontier models only achieve 65% constraint satisfaction, with feasibility being a bigger challenge than optimality.

• →ConstraintBench evaluates LLMs on direct constrained optimization across 10 operations research domains with Gurobi-verified solutions.
• →The best performing model achieved only 65% constraint satisfaction, indicating significant limitations in LLM reasoning capabilities.
• →No model exceeded 30.5% on joint feasibility and optimality within 0.1% of solver reference solutions.
• →Performance varies dramatically by domain, from 83.3% feasibility in production mix to just 0.8% in crew assignment.
• →Common failure modes include duration constraint misunderstanding and entity hallucination in complex scenarios.