QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

Researchers introduce QMFOL, an automated framework for generating controlled-complexity logical reasoning benchmarks to evaluate large language models. The resulting QMFOLBench dataset of 2,880 instances reveals that LLM reasoning performance degrades significantly with increased logical complexity, with models showing consistent bias toward true-labeled tasks over false or unknown ones.

QMFOL addresses a critical gap in AI evaluation methodology. As large language models become increasingly deployed in high-stakes domains requiring rigorous reasoning, existing benchmarks fail to systematically measure how models handle varying levels of logical complexity. This research contribution matters because it provides the first automated, scalable approach to generating reasoning tasks with precise control over difficulty parameters—a capability previously unavailable to the evaluation community.

The framework's innovation lies in its ability to construct formal logical structures with quantifiable complexity, then verify logical consistency through round-trip validation using external provers. This eliminates a major challenge in benchmark design: ensuring that natural language translations of logical tasks maintain their formal properties. Prior benchmarks either lacked this verification or relied on manual curation, limiting scale and reproducibility.

For the AI development industry, QMFOL's findings have immediate implications. The observed performance degradation with complexity suggests current models lack robust reasoning capabilities despite marketing claims. The discovered bias toward true-labeled tasks indicates systematic weaknesses that developers should address. These insights enable more precise model comparison and help organizations assess whether deployed models meet reasoning requirements for critical applications.

Looking forward, this framework could become standard infrastructure for reasoning evaluation, similar to how ImageNet transformed computer vision research. The 2,880-instance benchmark is modest but the automated generation approach scales indefinitely. Researchers should monitor whether subsequent model iterations improve on the identified complexity thresholds and whether the semantic sensitivity findings drive architectural innovations in reasoning-focused model design.

• →QMFOL enables precise control over logical complexity in reasoning benchmarks, addressing limitations of existing evaluation datasets.
• →Six reasoning models and two LLMs show measurable performance degradation as logical complexity increases, indicating capacity limits.
• →Models consistently perform better on true-labeled tasks than false or unknown ones, revealing systematic evaluation biases.
• →Round-trip verification with external provers ensures logical consistency between formal structures and natural language translations.
• →Automated generation framework scales indefinitely and could become standard infrastructure for reasoning capability evaluation.