Benchmarking Counterfactual Prediction in Epidemic Time Series with Time-Varying Interventions

Researchers have developed a large-scale benchmark dataset for evaluating causal inference methods in epidemic time-series prediction under dynamic interventions. Using calibrated agent-based models grounded in real-world U.S. county data, the benchmark enables testing of causal inference techniques across static and time-varying treatment scenarios with verifiable counterfactual outcomes.

This research addresses a fundamental challenge in causal machine learning: the scarcity of realistic datasets with ground-truth counterfactual outcomes. Traditional benchmarks either sacrifice realism by using simplified simulations or lack verifiable counterfactuals when drawn from real-world data. The authors bridge this gap by constructing a synthetic yet realistic benchmark grounded in actual demographic, mobility, epidemiological, and policy data from over 150 U.S. counties.

The significance stems from the benchmark's comprehensive scope. Unlike prior work supporting only static interventions, this framework accommodates time-varying treatments and multi-policy scenarios—reflecting real-world complexity where interventions change over time. This capability is crucial for developing robust causal inference methods applicable to dynamic policy environments, whether in pandemic response or other domains.

For the AI research community, this benchmark enables rigorous evaluation of causal inference algorithms at scale, moving beyond toy problems toward realistic assessment. The authors' evaluation of existing methods reveals substantial performance gaps, suggesting current approaches may struggle with practical epidemic forecasting tasks. This finding has implications for deploying causal AI systems in critical domains where accurate counterfactual reasoning directly informs policy decisions.

The work establishes a foundation for advancing causal time-series methods, particularly in epidemiology. Future research will likely build on this benchmark to develop more sophisticated causal inference techniques. The availability of realistic counterfactual trajectories enables faster iteration and more confident deployment of causal models in public health applications.

• →Researchers created a large-scale benchmark with realistic counterfactual outcomes across 150+ U.S. counties using calibrated agent-based epidemic models.
• →The benchmark uniquely supports both static and time-varying interventions, enabling evaluation across diverse causal inference scenarios.
• →Evaluation reveals substantial performance differences among causal inference methods, exposing challenges in realistic time-series causal reasoning.
• →The dataset grounds synthetic data in real demographic, mobility, epidemiological, and policy information for enhanced realism.
• →This benchmark addresses a critical gap enabling rigorous development of causal AI for dynamic policy environments.