Feature Starvation as Geometric Instability in Sparse Autoencoders

Researchers propose Adaptive Elastic Net Sparse Autoencoders (AEN-SAEs) to solve feature starvation in neural network interpretability tools. The method combines L2 and adaptive L1 regularization to create a mathematically stable sparse coding system that improves feature extraction in large language models without requiring complex workarounds.

This research addresses a fundamental problem in neural network interpretability: sparse autoencoders (SAEs) frequently develop dead neurons and biased representations when attempting to decompose the complex internal states of large language models into interpretable components. The standard approach using L1 regularization creates geometric instability that causes features to either starve or suffer from shrinkage bias, requiring practitioners to apply expensive computational fixes and non-differentiable masking techniques.

The core contribution lies in recognizing feature starvation as a structural mathematical problem rather than merely an artifact of training data. By introducing elastic net regularization with adaptive L1 reweighting, the researchers create a Lipschitz-continuous sparse coding map that maintains numerical stability throughout the encoding process. This approach combines the convexity properties of L2 regularization with intelligent L1 weighting that suppresses spurious features while preserving genuine signal.

The implications extend beyond academic theory into practical AI development. Better interpretability tools directly support safety research, model debugging, and feature attribution analysis—critical for understanding and auditing large language models. The method's fully differentiable architecture eliminates computational overhead compared to existing heuristic solutions, making SAE-based interpretability more accessible to researchers with standard computational resources.

For the AI research community, this work reduces friction in interpretability workflows and provides theoretical grounding for sparse representation learning. As organizations increasingly prioritize model transparency and safety, tools that efficiently extract interpretable features from opaque neural networks become strategically valuable. The empirical validation across multiple model scales (Pythia 70M through Llama 3.1 8B) demonstrates practical applicability rather than theoretical novelty alone.

• →Feature starvation in sparse autoencoders stems from fundamental geometric instability of L1 regularization, not just data issues
• →Adaptive elastic net SAEs achieve Lipschitz stability by combining L2 convexity with adaptive L1 reweighting
• →The fully differentiable approach eliminates need for expensive heuristic resampling and non-differentiable masking
• →Method successfully scales across LLMs from 70M to 8B parameters while maintaining competitive reconstruction
• →Improved interpretability tools support AI safety research and model auditing at reduced computational cost