Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decomposition

Researchers introduce Hierarchical Attribution Graph Decomposition (HAGD), a novel method for extracting sparse circuits from billion-parameter language models that reduces computational complexity from exponential to polynomial time. The approach successfully identifies interpretable pathways in models ranging from GPT-2 to Llama-70B, achieving 91% behavioral preservation on modular arithmetic tasks while existing methods like ACDC become memory-prohibitive at 1.4B parameters.

HAGD represents a significant advancement in mechanistic interpretability of large language models by solving a fundamental scalability problem. Previous circuit extraction methods faced exponential search spaces, making analysis infeasible beyond smaller models. This work compresses that search into O(n² log n) complexity through four stages: monosemantic transcoder training, spectral graph clustering, GNN-guided traversal, and causal verification. The method demonstrates robust performance across diverse model families and tasks, from arithmetic operations to commonsense reasoning benchmarks.

The research addresses a critical gap in AI transparency. As language models scale to hundreds of billions of parameters, understanding their internal decision-making becomes increasingly important for safety and reliability. HAGD's cross-architecture transfer coefficients (0.38-0.82) suggest circuits maintain structural consistency across model variants, opening possibilities for transferable interpretability insights.

For the broader AI development community, this work enables researchers to analyze increasingly large models without prohibitive computational costs. The method's successful application to 70B-parameter Llama models suggests it may scale further. However, significant limitations remain: the approach omits attention-head circuits, explains only 80-85% of reconstruction variance, and struggles with circuits exceeding several hundred nodes. These gaps indicate interpretability remains incomplete even with improved methods.

Looking forward, resolving the unexplained variance and extending analysis to attention mechanisms would strengthen the framework. The ability to transfer circuit understanding across model families could accelerate research into model behavior prediction and safer alignment techniques.

• →HAGD reduces circuit extraction complexity from exponential to O(n² log n), enabling analysis of billion-parameter models where existing methods fail
• →The method achieves 91% behavioral preservation on modular arithmetic with circuits containing only 49-347 nodes across various model sizes
• →Cross-architecture transfer coefficients reach 0.82 between Llama variants, suggesting circuits have consistent structure across related models
• →Current limitations include 15-20% unexplained reconstruction variance and inability to interpret circuits exceeding several hundred nodes
• →ACDC baseline becomes memory-prohibitive beyond 1.4B parameters while HAGD successfully processes models up to 70B parameters