HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

Researchers introduce HOLMES, a new benchmark for evaluating higher-order logical reasoning in large language models, revealing that current LLMs struggle significantly with complex symbolic reasoning tasks that go beyond simple first-order logic. The benchmark demonstrates critical gaps in AI reliability, with the best-performing models achieving only 59.54% accuracy on tasks involving reasoning over rules, predicates, and constraints across legal and financial domains.

HOLMES addresses a fundamental weakness in how AI systems are evaluated and trained. While existing benchmarks focus on first-order logic—straightforward deductions about objects and relationships—real-world reasoning often requires higher-order thinking where models must reason about rules themselves, manipulate predicates, and navigate complex constraints. The benchmark's findings expose a critical vulnerability: LLMs can achieve high accuracy on final answers while employing shortcuts that lack genuine logical rigor, particularly problematic in domains like law and finance where verifiable reasoning is essential.

This research builds on growing concerns about AI reliability in high-stakes applications. As organizations integrate LLMs into decision-making systems, the ability to verify reasoning chains becomes increasingly important. The sharp performance drops under compositional and scope-conditioned reasoning suggest LLMs fail at systematic logical composition, a fundamental cognitive capability. The public release of HOLMES and its 1,379 annotated instances provides the AI research community with concrete tools to identify and address these shortcomings.

The implications extend beyond academic research. Developers building AI systems for legal contracts, financial analysis, and compliance face a clear constraint: current LLMs cannot be trusted for tasks requiring rigorous symbolic reasoning without significant oversight. This creates opportunities for specialized tools and human-AI collaboration frameworks. Organizations planning to deploy LLMs in regulated industries must account for these limitations, likely increasing demand for interpretability tools, formal verification methods, and human review processes.

• →Current LLMs achieve only 50.64% average accuracy on higher-order logical reasoning tasks, with best models reaching 59.54%.
• →High final-answer accuracy can mask flawed reasoning shortcuts, creating false confidence in AI system reliability.
• →Performance collapses under compositional and scope-conditioned reasoning, revealing fundamental limitations in logical thinking.
• →HOLMES dataset covers law and finance domains with verifiable reasoning traces, enabling targeted AI improvement research.
• →The benchmark identifies higher-order symbolic reasoning as a critical bottleneck for deploying reliable LLMs in regulated industries.