Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability

Researchers demonstrate that standard Sparse Autoencoders (SAEs) used for interpreting large language models suffer from a fundamental architectural flaw: their single-direction decoders cannot efficiently represent multi-dimensional features, causing unnecessary feature splitting. They introduce Subspace-Aware Sparse Autoencoders (SASA) with learned decoder subspaces that reduce this splitting while achieving better interpretability and monosemanticity on GPT-2 and Mistral-7B with half the training tokens.

This research addresses a critical technical problem in mechanistic interpretability, a field gaining prominence as AI systems become more opaque. SAEs have become standard tools for decomposing neural network activations into interpretable components, but the paper reveals they operate under a flawed assumption: treating all features as one-dimensional entities. This mismatch forces the model to split coherent multi-dimensional features across numerous near-collinear latents, obscuring the actual feature geometry and creating spurious interpretations.

The theoretical contribution is substantial. The authors prove that reconstructing a d-dimensional feature with single-direction decoders requires exponentially many atoms—a geometric impossibility that current training objectives actively prefer. They demonstrate a continuous optimization path showing why standard SAEs inherently drive toward feature fragmentation, not because of implementation details but because the loss function itself encourages this behavior.

SASA's solution elegantly addresses this by replacing single decoders with learned subspaces, enforcing block sparsity, and adaptively controlling effective rank. Critically, they prove that with appropriate block sizes, a single group becomes the global minimizer—converting the exponential sample complexity problem into polynomial complexity. In practical terms, this means drastically fewer required LLM forward passes during training.

The empirical validation on real models (GPT-2 and Mistral-7B) shows SASA simultaneously improves monosemanticity, reduces spurious features, and reduces computational requirements by half. For the mechanistic interpretability community, this represents a significant advance in understanding what SAEs actually capture and how to extract cleaner feature decompositions. The work likely influences future interpretability research directions and tool development.

• →Standard SAEs' single-direction decoders provably cause exponential feature splitting for multi-dimensional features through both geometric and optimization-theoretic mechanisms.
• →SASA replaces single-vector decoders with learned subspaces, reducing sample complexity from exponential to polynomial in feature dimension.
• →Empirical results show SASA improves monosemanticity and interpretability while requiring approximately 50% fewer training tokens than standard SAEs.
• →The research demonstrates that current SAE interpretability results may be contaminated by artifacts caused by architecture-induced feature fragmentation.
• →The theoretical framework provides formal guarantees that well-configured SASA groups become global minimizers of the training objective.