TempoBench: Evaluating Temporal Causal Reasoning in Large Language Models

Researchers introduce TempoBench, a formally verified benchmark for evaluating temporal causal reasoning in large language models, revealing a significant gap between forward simulation performance (96% accuracy) and causal reasoning ability (below 25%). The study demonstrates that LLMs struggle with identifying minimal causal inputs, instead over-specifying by listing all possible inputs rather than reasoning about necessity.

TempoBench addresses a fundamental limitation in how current large language models approach reasoning tasks. While LLMs excel at forward simulation—predicting outcomes from given inputs—they systematically fail at the inverse problem of temporal causal reasoning: determining which minimal set of prior inputs necessarily caused an observed result. This distinction carries significant implications for AI safety, interpretability, and deployment in domains requiring causal accountability.

The research stems from growing recognition that benchmark performance alone masks critical reasoning deficiencies. Traditional evaluations often reward surface-level pattern matching over genuine causal understanding. By constructing a formally verified benchmark using Mealy machines with provably correct labels, the researchers created a gold standard for evaluation. The finding that 94% of errors involve overspecification—where models retrieve and enumerate all possibilities rather than filtering for necessity—suggests a fundamental architectural or training limitation rather than simple capability gaps.

This work affects AI developers building reasoning-dependent systems and organizations deploying LLMs for decision support. Models trained on TempoBench show improved generalization across standard reasoning benchmarks, indicating that targeted causal reasoning training enhances broader capabilities. The gap between simulation and causal reasoning performance raises questions about whether current scaling approaches adequately develop genuine reasoning capacity or merely improve pattern recognition.

Future development hinges on whether fine-tuning improvements represent temporary gains or signal a path toward more robust causal reasoning. The benchmark itself provides infrastructure for measuring progress on a previously unquantified problem, potentially catalyzing focused research into causal reasoning mechanisms in neural networks.

• →LLMs achieve 96% accuracy on temporal simulation but drop below 25% on minimal causal attribution tasks.
• →94% of causal reasoning errors stem from overspecification, where models list all inputs instead of identifying necessary causes.
• →TempoBench provides the first formally verified benchmark for evaluating temporal causal reasoning with provably correct labels.
• →Fine-tuning on TempoBench training data improves causal reasoning and generalizes better than math or code-based training.
• →The research reveals a critical gap between forward prediction capabilities and inverse causal reasoning in frontier LLMs.