Steering Vision-Language Models with Joint Sparse Autoencoders

Researchers introduce Joint Sparse Autoencoders (JSAE), a technique that improves how vision-language models can be analyzed and controlled by aligning visual and textual representations into shared, interpretable features. Testing across multiple VLM architectures reveals that steering interventions work most effectively at mid-to-late layers, offering insights for more precise multimodal model control.

This research addresses a fundamental challenge in multimodal AI: understanding and controlling how vision-language models process information across visual and linguistic domains. Sparse Autoencoders have proven useful for interpreting individual modalities, but applying them to cross-modal systems historically produced muddled representations that resisted practical steering. JSAE introduces an explicit alignment constraint that forces the model to find shared feature representations across both vision and language pathways simultaneously, yielding interpretable concepts like 'food' and 'animals' that correspond to real-world categories.

The technical contribution matters because interpretability and controllability of large multimodal models remain open problems as these systems become increasingly deployed in production environments. Previous work on SAEs focused on language-only models; extending this to VLMs requires handling two distinct activation streams. The layer-dependent asymmetry findings—where additive steering peaks in mid-to-late layers while suppression remains consistent—suggest that information flow through multimodal models follows predictable architectural patterns that developers can exploit.

For the broader AI field, this work provides practical tools for scientists and engineers building safer, more controllable multimodal systems. The consistency of results across three different VLM architectures (including a mixture-of-experts variant) indicates the findings generalize beyond single implementations. Researchers in mechanistic interpretability and AI safety will find this especially relevant for understanding how to intervene on model behavior without full retraining. The layer-localized effects open new research directions into how vision and language information interacts at different depths of neural networks.

• →Joint Sparse Autoencoders enable recovery of interpretable cross-modal features in vision-language models through explicit alignment constraints.
• →Additive steering interventions work most effectively at mid-to-late model layers, while suppression effects remain consistent across layers.
• →The approach successfully identifies recognizable concepts like food and animals as shared representations between visual and textual modalities.
• →Results replicate consistently across three different VLM architectures, suggesting findings generalize beyond individual model designs.
• →Better multimodal steering capabilities enable improved safety analysis and control mechanisms for vision-language models.