Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers

Researchers demonstrate that attention heads in large language models passing standard mechanistic interpretability tests—necessity, linear encoding, and ablation recovery—fail to transfer their computations to different contexts. The study introduces KID framework and a three-stage validation pipeline, revealing that many claimed attention head roles are artifacts of specific prompt contexts rather than genuine semantic functions.

This research challenges a foundational assumption in mechanistic interpretability: that attention heads with clear behavioral signatures possess consistent, transferable computational roles. The authors tested this hypothesis across three 7-8B instruction-tuned models and five computation families, finding that heads meeting all traditional validation criteria routinely failed when their activations were patched into different prompts under controlled conditions. This gap between necessity and transferability suggests that mechanistic interpretability studies may be overestimating how well they understand model internals.

The work builds on growing skepticism about whether attention head role assignments reflect genuine model mechanisms or merely correlations within specific experimental contexts. Previous research hinted at such limitations, but this study provides systematic evidence across multiple models and computation types. The introduction of the KID framework—distinguishing between knowing (encoding information), intent (necessity for behavior), and doing (transfer capability)—offers a more rigorous taxonomy for evaluating mechanistic claims.

For AI researchers and safety teams relying on mechanistic interpretability to understand model behavior, this finding carries significant implications. If current validation methods systematically overstate our understanding of how models work, then interpretability-based safety arguments may rest on shakier ground. The same-answer control approach the authors highlight—comparing computations against alternatives that reach identical outputs through different reasoning—represents a valuable methodological refinement that could prevent false claims of mechanistic understanding from propagating through the literature.

• →Attention heads passing standard mechanistic interpretability tests fail to transfer computations across different prompts, indicating insufficient validation methods.
• →The KID framework distinguishes knowing, intent, and doing as separate dimensions for assigning roles to attention heads.
• →Same-answer controls expose state transfer being mistaken for semantic specificity in mechanistic interpretability research.
• →Current role claims about attention heads may reflect prompt-specific artifacts rather than genuine computational mechanisms.
• →Mechanistic interpretability-based arguments for AI safety require more rigorous validation before drawing strong conclusions.