Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics

Researchers applied mechanistic interpretability techniques to Walrus, a foundation model for continuum dynamics, using sparse autoencoders to probe internal mechanisms. The study reveals inconsistent feature alignment with known physics and systematic discrepancies in model outputs, highlighting fundamental challenges in understanding and validating scientific AI systems.

This research addresses a critical gap in AI validation for scientific domains where ground-truth physics is well-established. Unlike general-purpose AI systems evaluated primarily on benchmark performance, scientific foundation models must demonstrate that their predictions arise from physically plausible internal mechanisms. The Walrus study reveals a sobering reality: even when a model reproduces empirical results accurately, its internal representations may not correspond to known physics in predictable or interpretable ways.

The mechanistic interpretability approach using sparse autoencoders represents a rigorous methodology for probing AI black boxes in scientific contexts. By analyzing over 20,000 features through the lens of enstrophy—a physically meaningful metric—the researchers employed domain expertise to triage mechanistic analysis. Their finding of "piecewise consistency" suggests that features recur in similar roles across setups, yet this structure remains intermittent and cannot be cleanly mapped to standard physical decompositions.

These findings carry implications for the broader adoption of AI emulators in scientific computing. When foundation models fail—producing diffuse or overly localized structures—the connection to specific feature usage provides diagnostic value. However, the work reveals that single-layer analysis and SAE limitations create analysis artifacts that complicate interpretation. This suggests current interpretability tools may be insufficient for validating scientific models in high-stakes domains.

The research underscores that model effectiveness in reproducing data does not guarantee mechanistic transparency. The open questions posed—prioritizing meaningful features, distinguishing stable structure from artifacts, and determining when different representations are informative—will likely shape validation standards for scientific AI systems moving forward.

• →Foundation models can accurately reproduce physical dynamics while maintaining internally inconsistent or non-physical mechanisms.
• →Sparse autoencoders reveal intermittent feature reuse across simulation setups, but this structure does not map cleanly to known physics.
• →Output-level discrepancies in energy distribution correlate with changes in specific feature usage patterns.
• →Current mechanistic interpretability tools have significant limitations in distinguishing meaningful structure from analysis artifacts.
• →Scientific foundation models require new validation frameworks beyond performance benchmarks to ensure physical plausibility.