Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformers

Researchers introduce ViSAE, a mechanistic interpretability toolbox that uses neuroscience-inspired principles to decode how Vision Transformers make decisions through human-interpretable concept circuits. The method achieves significant improvements in model auditing and steering, with concept editing improving worst-group accuracy by 48.2% on benchmark tests, addressing critical safety concerns before ViT deployment.

Vision Transformers have achieved impressive accuracy rates across computer vision tasks, yet their decision-making processes remain opaque—a critical liability when deployed in safety-critical applications. ViSAE addresses this interpretability gap by leveraging sparse autoencoders to decompose ViT representations into understandable concepts, drawing inspiration from how neuroscience explains biological vision processing. The toolbox introduces a substantially improved concept vocabulary of 16K visually grounded terms and a probing suite of 64K images, delivering 20x better concept coverage efficiency and 28.7% higher interpretation accuracy compared to existing approaches.

The development reflects growing industry recognition that neural network transparency is essential for responsible AI deployment. Current methods for interpreting transformer models remain limited by subjective feature analysis and inconsistent concept coverage, creating blind spots in model auditing. ViSAE's top-down concept reading and bottom-up circuit tracing algorithms automate the discovery of internal decision pathways, moving beyond manual inspection toward systematic understanding of model behavior.

The practical impact becomes evident in ViSAE's steering capabilities. By editing specific concepts, researchers achieved 48.2% improvement in worst-group accuracy on the WaterBirds dataset—a 23.8% performance advantage over existing debiasing methods. This directly addresses spurious correlation problems where models make correct predictions for wrong reasons. For developers and organizations deploying vision systems, ViSAE provides a framework for auditing model reliability before production release, reducing risks from latent biases.

Looking forward, this work establishes mechanistic interpretability as a practical tool rather than purely theoretical exercise. Broader adoption of such interpretability frameworks could become a standard requirement for high-stakes vision model deployments, particularly in autonomous systems and medical imaging applications.

• →ViSAE enables automated discovery of Vision Transformer decision circuits using neuroscience-inspired mechanistic interpretability techniques.
• →The method achieves 20x better concept coverage efficiency and 28.7% higher interpretation accuracy versus existing concept-based approaches.
• →Concept editing via ViSAE improves worst-group accuracy by 48.2%, outperforming other debiasing methods by 23.8%.
• →The toolbox provides practical tools for model auditing and steering, addressing safety concerns before ViT deployment.
• →Code and datasets are publicly available, enabling broader adoption of interpretability practices in vision model development.