LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

Researchers introduce LGMT, a novel testing framework that uses first-order logic to evaluate Large Language Models' reasoning reliability by creating logically equivalent test cases. The study reveals that state-of-the-art LLMs fail consistency checks under semantic transformations, exposing hidden reasoning defects that traditional benchmarks miss.

The evaluation of Large Language Models has relied heavily on static benchmarks that often provide an inflated view of reasoning capabilities. LGMT addresses this fundamental gap by applying metamorphic testing—a technique borrowed from software engineering—to assess LLM robustness under logically equivalent transformations. Rather than comparing outputs against reference answers, the framework checks whether models maintain consistent reasoning across semantically invariant cases derived from formal logic principles.

This research emerges amid growing scrutiny of LLM reasoning abilities. While models demonstrate strong performance on benchmark tasks, their actual robustness in production environments remains questioned. The paper's findings that advanced prompting techniques like Few-shot Chain-of-Thought only partially mitigate reasoning failures suggests the limitations are more fundamental than previously assumed. Symbol-level and conclusion-level variations prove particularly challenging for current architectures.

The implications extend across AI development and deployment. Organizations relying on LLMs for critical reasoning tasks face uncertainty about true model capabilities. This framework provides a diagnostic tool that could shape how companies evaluate and select models. For AI researchers, these results indicate that architectural improvements addressing logical invariance, rather than benchmark optimization, may be necessary for genuine reasoning advancement.

The work establishes a methodological foundation for more rigorous LLM evaluation. As the field moves toward deploying these systems in high-stakes applications, moving beyond isolated correctness measurements toward robustness validation becomes essential. Future research should explore whether the identified defects correlate with specific model architectures or training approaches, and whether targeted fine-tuning can improve performance on logically invariant tests.

• →LGMT exposes hidden reasoning defects in state-of-the-art LLMs that static benchmarks fail to detect through logically equivalent test transformations
• →Current LLMs demonstrate significant inconsistency when facing symbol-level and conclusion-level variations despite strong benchmark performance
• →Few-shot Chain-of-Thought prompting only partially addresses reasoning vulnerabilities, suggesting deeper architectural limitations
• →The framework shifts LLM evaluation from isolated correctness toward robustness under logical invariance principles
• →Oracle-free metamorphic testing provides a scalable, principled approach for diagnosing and tracking reasoning failures across model iterations