Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers

Researchers present a three-step methodology for identifying and validating attention-head circuits in transformer models using spectral analysis, pattern filtering, and causal ablation. The technique successfully isolates core computational circuits across multiple model sizes and architectures without requiring labeled data or gradient attribution.

This research addresses a fundamental challenge in transformer interpretability: understanding which neural components perform specific computational tasks and why they matter. The spectral probe-circuits methodology offers a reproducible framework that moves beyond observational analysis toward causal claims about model behavior. By using time-integrated participation ratios as a ranking signal, researchers eliminate the need for supervised labels, enabling discovery of model-specific circuits across diverse architectures and training regimes.

The work builds on growing interest in mechanistic interpretability within AI research, where scientists seek to understand transformer internals rather than treating them as black boxes. Previous approaches relied on gradient-based attribution methods that can be noisy or computationally expensive. This three-step recipe—spectral ranking, task-pattern screening, and matched-random ablation controls—represents a more efficient pathway to circuit identification.

The validation spans significant parameter ranges (51M to 7B parameters) and includes both dense transformers and mixture-of-experts variants, demonstrating generalizability. The finding that induction circuits remain consistently small (3-11 heads) despite scaling to larger models suggests fundamental computational constraints. The observation that only 17-19% of heads perform identifiable specialized computation indicates substantial redundancy in neural networks, relevant for both efficiency and robustness research.

These insights have implications for model compression, training efficiency, and safety evaluation. Understanding circuit structure enables targeted interventions and more precise model debugging. The methodology's reproducibility across independent seeds suggests it captures genuine model properties rather than artifacts, strengthening confidence in mechanistic interpretability as a discipline.

• →A spectral-based method identifies attention circuits without labels or gradient attribution, using time-integrated participation ratios to rank functional heads.
• →Induction circuits of 2-6 heads appear causally necessary in all tested models, causing 94-100% performance drops when ablated.
• →Only 17-19% of transformer heads perform identifiable specialized computation, revealing substantial functional redundancy across scales.
• →The methodology generalizes across parameter ranges from 51M to 7B models and multiple architecture families including dense and mixture-of-experts variants.
• →Circuit identification is reproducible across independent model seeds without supervision, indicating the method captures genuine computational structures.