Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models

Researchers demonstrate that identical mechanistic identification recipes for neural circuit analysis produce inconsistent results across different language model architectures, revealing that the same task capability is implemented through fundamentally different attention patterns in models from distinct training pipelines. This finding challenges assumptions about universal mechanistic explanations in AI systems and introduces a taxonomy for circuit screening outcomes.

This mechanistic study exposes a critical gap between methodological consistency and substantive reproducibility in AI interpretability research. The authors applied a standardized screen-and-ablate protocol across three 1B-class language models performing four composed tasks, finding that identical behavioral capabilities emerge from entirely different attention-head circuits depending on model architecture and training pipeline. This divergence matters because the field has increasingly relied on identifying mechanistic explanations to understand model behavior, but the study suggests these explanations may reflect implementation details rather than fundamental task structures.

The research emerges from growing interest in mechanistic interpretability—the effort to reverse-engineer neural networks at the circuit level. Previous studies identified specific attention patterns implementing tasks like indirect-object identification or greater-than comparisons, implying these patterns were causal necessities. This work challenges that interpretation, proposing instead that different model families develop functionally equivalent but structurally distinct solutions.

For the AI research community, this creates both methodological and conceptual implications. The finding that mixture-of-experts (MoE) models build circuits on previous-token positional substrates offers one concrete insight, but the broader takeaway constrains mechanistic claims' generalizability. Researchers and practitioners relying on circuit-level explanations must now account for model-specific implementations when drawing conclusions.

Moving forward, the field needs either architectural-invariant mechanistic theories or explicit acknowledgment that circuits are implementation-specific artifacts. The introduced taxonomy provides structure for categorizing screening outcomes, establishing clearer standards for interpretability claims across diverse model architectures.

• →Identical mechanistic identification methods produce different circuit explanations across model architectures, challenging universality assumptions in neural circuit analysis
• →The same task capability emerges from structurally distinct attention patterns in Pythia, OLMo, and OLMoE 1B models despite identical behavioral performance
• →MoE models appear to build composed-task circuits on top of previous-token positional substrates, suggesting architecture-specific implementation strategies
• →A five-category screening taxonomy distinguishes primary causes, secondary causes, correlates, interferers, and nulls in mechanistic analysis
• →Mechanistic explanations may reflect model-specific implementation details rather than fundamental task structures required for capability emergence