Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression

Researchers introduce a novel semantic distance metric for sparse autoencoders (SAEs) using distributional representations and Wasserstein distance, enabling better cross-layer feature matching and automatic circuit compression in language model interpretability research.

This research addresses a critical bottleneck in mechanistic interpretability of large language models. Sparse autoencoders have emerged as a promising tool for understanding how neural networks process information, but scaling these analyses across multiple layers has remained computationally and methodologically challenging. The paper's key innovation reframes feature matching as a distributional comparison problem rather than simple vector alignment, representing each SAE feature as an activation-weighted distribution over hidden states rather than a single decoder vector.

The distributional approach represents meaningful progress in interpretability research. By projecting distributions into shared reference spaces and using Wasserstein distance—a mathematically robust metric for comparing probability distributions—the authors create a unified framework that handles previously intractable problems. The theoretical guarantees around invariance to activation rescaling and stability under perturbations provide confidence in the method's robustness. These properties matter because SAE features often exhibit numerical instabilities that break simpler approaches.

The broader implications extend beyond academic interpretability. As AI safety and regulatory scrutiny intensify, the ability to mechanistically understand model behavior becomes increasingly valuable. Better feature matching and circuit compression tools could accelerate efforts to identify potential failure modes, verify alignment properties, and build trust in AI systems. Automated supernoding capability particularly stands out—reducing complex feature circuits into interpretable components directly supports the goal of explainable AI systems.

Looking ahead, adoption depends on whether this method integrates into existing interpretability frameworks and scales to the latest model architectures. The research community will likely test these techniques on increasingly large models and compare results with competing interpretability approaches.

• →Distributional representations of SAE features enable cross-layer matching without requiring vector alignment across different activation manifolds.
• →Wasserstein distance provides mathematically principled comparison of semantic similarity with theoretical guarantees on invariance and stability.
• →Method automatically compresses large feature circuits into interpretable supernodes, accelerating circuit analysis workflows.
• →Theoretical framework proves robustness to activation rescaling, addressing a fundamental challenge in current SAE comparison approaches.
• →Empirical results outperform decoder-vector and LLM-based baselines while capturing subtle functional distinctions between related features.