Mathematical Reasoning via Intervention-Based Time-Series Causal Discovery Using LLMs as Concept Mastery Simulators

Researchers propose CIKA, a framework using LLMs as interventional simulators to identify which mathematical concepts causally contribute to correct answers, distinguishing genuine causal relationships from spurious correlations. The method achieves 69.7% on Omni-MATH-Rule and 97.2% on GSM8K with a frozen 7B model, outperforming o1-mini on contamination-free benchmarks.

This research addresses a fundamental challenge in AI reasoning systems: determining which concepts actually enable problem-solving versus merely correlating with correct answers. Traditional observational methods cannot distinguish causal relationships from confounding factors like problem difficulty, limiting their ability to diagnose genuine capability gaps. CIKA solves this by treating the LLM itself as an interventional simulator, using prompts to artificially set concept mastery states and measuring resulting performance changes.

The work builds on growing recognition that LLMs often possess knowledge they fail to activate in appropriate contexts. Previous approaches using Monte Carlo Tree Search or causal graph injection lacked mechanisms to isolate causal contributions. By formalizing concept causality through Interventional Capability Probes (ICP), the authors create measurable causal effects independent of confounding variables. Their statistical validation on 67 problems demonstrates ICPs discriminate relevant from irrelevant concepts with high confidence (p < 10^-6).

The practical implications extend beyond academic interest. A frozen 7B parameter model achieving 69.7% on Omni-MATH-Rule—outperforming OpenAI's o1-mini at 60.5%—suggests efficient reasoning improvements without large-scale training or parameter updates. The finding that 33.8% of additionally correct answers come from activating dormant knowledge rather than learning new concepts indicates development resources could prioritize knowledge activation mechanisms.

Future applications likely involve integrating causal knowledge diagnostics into training pipelines to systematically improve reasoning. The framework's ability to identify concept bottlenecks could accelerate development of specialized reasoning systems across domains requiring multi-step causal chains.

• →CIKA framework uses interventional probes to identify causally relevant mathematical concepts, distinguishing genuine capabilities from spurious correlations.
• →A frozen 7B LLM outperforms o1-mini (69.7% vs 60.5%) on contamination-free benchmarks using knowledge activation rather than new learning.
• →Interventional Capability Probes show statistically significant discrimination between causally relevant and irrelevant concepts (p < 10^-6).
• →Analysis reveals 33.8% of additional correct answers result from activating existing knowledge rather than learning new concepts.
• →Solved problems demonstrate 6.1× higher average treatment effects than unsolved ones, confirming ICP predictability for problem-solving success.