Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines

A new study challenges recent findings that dismissed Sparse Autoencoders (SAEs) as ineffective for steering Large Language Models, demonstrating that SAEs can match LoRA baseline performance when combined with a supervised feature selection pipeline. The research suggests that high sparsity constraints may not be necessary for effective model steering based on interpretability.

The debate over Sparse Autoencoders' utility for LLM steering reflects the ongoing challenge of understanding and controlling neural network behavior. Wu et al. (2025) initially concluded that SAEs underperformed simple baselines on the AxBench benchmark, casting doubt on this interpretability approach. This new work contests that conclusion by introducing a supervised pipeline for feature selection and labeling that enables SAEs to achieve competitive performance with reference LoRA methods.

The significance lies in how researchers approach interpretability and control mechanisms for LLMs. Sparse Autoencoders decompose model activations into interpretable features, theoretically offering transparency into what mechanisms drive model behavior. If SAEs can reliably steer outputs while maintaining interpretability advantages, they become more valuable for both safety research and model understanding. The finding that causality emerges from interpretability-based components suggests the approach captures genuine model structure rather than spurious correlations.

For the AI research community, this work impacts the direction of future mechanistic interpretability work. If high sparsity constraints prove unnecessary, researchers may focus computational resources elsewhere. This broadens the practical applicability of SAEs across different model scales and architectures. The implications extend to AI safety efforts, where interpretable steering mechanisms are increasingly critical as models become more capable.

Future research should investigate whether this supervised pipeline generalizes across different model architectures, scales, and domains. The interaction between sparsity levels and steering effectiveness warrants deeper investigation, particularly whether the optimal sparsity varies by model type or steering objective.

• →Sparse Autoencoders can match LoRA performance on model steering when combined with supervised feature selection pipelines.
• →Interpretability-based components alone can identify features that causally influence model outputs.
• →High sparsity constraints may not be necessary for effective steering based on interpretability, contradicting earlier findings.
• →The results suggest SAEs were underestimated in previous benchmarks due to insufficient feature selection methodology.
• →Findings have implications for AI safety research and mechanistic interpretability approaches to LLM control.