Can Large Language Models Infer Causal Relationships from Real-World Text?

Researchers developed the first real-world benchmark for evaluating whether large language models can infer causal relationships from complex academic texts. The study reveals that LLMs struggle significantly with this task, with the best models achieving only 0.535 F1 scores, highlighting a critical gap in AI reasoning capabilities needed for AGI advancement.

This research addresses a fundamental limitation in current large language models: their ability to understand cause-and-effect relationships in naturally occurring text. Unlike previous studies relying on synthetic or simplified examples with explicit causal statements, this benchmark draws from real academic literature, incorporating varying text lengths, domain diversity, and different levels of causal complexity. This methodological shift is crucial because real-world reasoning demands inferring implicit causal relationships amid dense information and competing narratives.

The poor performance metrics—a maximum F1 score of 0.535—demonstrate that even leading LLMs operate at near-random performance levels on this task. This finding has significant implications for AI development trajectories. Causal reasoning represents a cornerstone of human cognition and is essential for systems approaching artificial general intelligence. Current LLMs excel at pattern matching and statistical inference but falter when required to construct logical causal chains in complex, nuanced texts.

For AI researchers and developers, this benchmark provides targeted diagnostic insights into specific failure modes: explicitness levels, event density, text length, and domain-specific challenges. Organizations building AI systems for research, policy analysis, or scientific literature review should recognize these limitations when deploying LLMs in high-stakes domains where causal understanding is critical.

Moving forward, the open-source release of this dataset creates opportunities for the research community to develop improved architectures and training methodologies. Progress here could accelerate LLM capabilities in reasoning tasks that require deeper semantic understanding beyond surface-level pattern recognition. The benchmark effectively establishes a new frontier for evaluating and advancing AI reasoning capabilities.

• →Current LLMs achieve only 0.535 F1 score on real-world causal inference tasks, indicating fundamental reasoning limitations.
• →The first-ever real-world causal reasoning benchmark reveals performance gaps that synthetic datasets could not expose.
• →Text complexity factors including length, explicitness, and domain diversity significantly impact LLM causal reasoning accuracy.
• →Causal relationship inference represents a critical capability gap between current AI and artificial general intelligence requirements.
• →Open-source dataset release enables focused research into improving LLM reasoning on semantically complex tasks.