Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures

Researchers present Causal Agent Replay (CAR), a new method for diagnosing why large language model agents fail by identifying which decision step caused a failure rather than just which action executed it. Using structural causal models and intervention-based analysis, CAR achieves significantly higher attribution accuracy than existing LLM-judge approaches and provides confidence-bounded explanations for agent failures.

LLM-based autonomous agents are increasingly deployed in high-stakes environments where failures can result in data breaches, incorrect transactions, or incorrect tool usage. Current debugging approaches answer surface-level questions—what happened or whether a test passed—but fail to pinpoint the causal origin of failures. This gap matters because the step that executes a harmful action often differs from the step that decided on it, making naive attribution unreliable. Existing state-of-the-art LLM-based judges achieve only ~14% accuracy on step-level attribution benchmarks, leaving practitioners without reliable tools to understand agent misbehavior.

The research builds on causal inference principles by modeling agent execution as a structural causal model and applying interventional analysis rather than correlational methods. The approach introduces an intervention algebra for agent steps and a single-step contrastive estimator that addresses confounding specific to stochastic forward execution. To handle complex interactions, the authors developed a budget-bounded Monte-Carlo Shapley estimator that distributes credit across multiple contributing steps. Validation against synthetic models with known ground truth shows the method recovers both single pivotal steps and two-step interactions with near-perfect efficiency (0.909 vs. analytic 0.91).

For AI developers and organizations deploying agentic systems, this work provides a practical tool for root-cause analysis that runs on open-source or free local models. The confidence-interval reporting enables teams to trust attribution results and prioritize debugging efforts. As LLM agents handle more critical workflows in customer service, finance, and healthcare, deterministic failure diagnosis becomes essential for safety and compliance. The open-source availability means adoption barriers are minimal, potentially accelerating broader deployment of debuggable agentic systems.

• →Causal Agent Replay identifies the decision step causing LLM agent failures, not just the execution step, improving attribution accuracy far beyond LLM-judge baselines.
• →The method uses structural causal models and interventional analysis to provide confidence-bounded explanations rather than unreliable correlational approaches.
• →Validation on synthetic models shows the approach recovers both single-step causes and multi-step interactions with 90%+ efficiency.
• →Open-source availability on hosted or local models removes deployment friction for organizations debugging agentic systems.
• →Reliable agent failure diagnosis becomes critical as LLM agents expand into high-stakes domains like finance, healthcare, and customer service.