Mechanistic Interpretability as Statistical Estimation: A Variance Analysis

Researchers demonstrate that mechanistic interpretability—the process of reverse-engineering AI model behaviors through circuit discovery—suffers from fundamental statistical instability due to high variance in causal mediation analysis. The findings reveal that circuit structures are fragile and highly sensitive to input data and hyperparameter changes, calling into question the scientific validity of existing MI methodologies and necessitating stricter statistical practices in the field.

This research addresses a critical vulnerability in mechanistic interpretability, a field gaining prominence as AI systems become more complex and opaque. The authors establish that circuit discovery, widely treated as a deterministic reverse-engineering task, is fundamentally a statistical estimation problem plagued by inherent variance. Single-input causal mediation analysis scores—the foundation of most circuit discovery pipelines—exhibit volatile results, meaning the causal effects attributed to neural network components are not fixed properties but unstable random variables.

The instability compounds through multiple layers of the discovery pipeline. Approximation methods like Edge Attribution Patching introduce additional noise, and aggregating these noisy estimates across datasets produces fragile structural conclusions. This creates a cascade effect where small perturbations in input data or hyperparameter selection yield dramatically different circuit structures, undermining reproducibility and scientific rigor.

For the AI research community, these findings represent a significant methodological correction. Currently, many MI papers present circuits as definitive discoveries without reporting stability metrics or quantifying variance sources. This work demands transparency about statistical uncertainty and robustness testing. Researchers face pressure to validate their findings against perturbations and report confidence intervals rather than point estimates.

The broader implications extend to AI safety and interpretability claims. If circuits are unstable artifacts rather than genuine functional sub-networks, interpretability efforts built on these discoveries may provide false confidence in understanding model behavior. Moving forward, the field must establish standardized stability metrics, adopt more rigorous statistical frameworks, and report confidence measures as routine practice to ensure mechanistic interpretability research maintains scientific validity.

• →Circuit discovery in neural networks exhibits fundamental statistical instability due to high variance in causal mediation analysis scores.
• →Approximation methods and dataset aggregation amplify variance, making discovered circuits sensitive to minor input or hyperparameter changes.
• →Current mechanistic interpretability research lacks adequate stability metrics and robustness reporting standards.
• →The scientific validity of existing MI findings is questionable without rigorous statistical analysis and uncertainty quantification.
• →The field must adopt mandatory stability testing and confidence interval reporting to establish credible interpretability claims.