ICA Lens: Interpreting Language Models Without Training Another Dictionary
Researchers introduce ICALens, a new method for interpreting language model representations using independent component analysis (ICA) instead of expensive sparse autoencoders (SAEs). The approach efficiently recovers interpretable directions without requiring large neural dictionary training, achieving competitive performance on standard benchmarks while offering a faster, more accessible alternative for LLM analysis.