Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

Researchers introduce Dictionary-Aligned Concept Control (DACO), a framework that uses a curated dictionary of 15,000 multimodal concepts and Sparse Autoencoders to improve safety in multimodal large language models by steering their activations at inference time. Testing across multiple models shows DACO significantly enhances safety performance while preserving general-purpose capabilities without requiring model retraining.

The development of DACO addresses a critical vulnerability in multimodal AI systems that process both text and images. Current safety approaches often struggle because they either require expensive retraining, depend on prompt engineering that fails against adaptive attacks, or affect unrelated model behaviors when targeting specific unsafe concepts. DACO's activation steering at inference time offers a computationally efficient alternative that works on frozen models.

This research emerges from the broader AI safety landscape where defenders continuously adapt to evolving attack patterns. The creation of DACO-400K, a dataset of 400,000 caption-image stimuli organized into 15,000 concept directions, represents substantial foundational work. By leveraging Sparse Autoencoders trained with the curated dictionary, researchers achieve granular control—adjusting specific harmful concepts without collateral damage to model outputs. The approach's compatibility with multiple architectures (QwenVL, LLaVA, InternVL) demonstrates transferability.

For the AI development community, this work establishes a new standard for practical safety interventions. Developers deploying multimodal models can now implement post-hoc safety measures without architectural modifications or expensive computational overhead. This democratizes safety implementation across organizations with varying resources. The framework's effectiveness across established benchmarks (MM-SafetyBench, JailBreakV) validates its real-world applicability.

Looking forward, the bottleneck involves scaling dictionary creation and ensuring concept coverage remains comprehensive as attackers discover novel exploitation vectors. The sparse autoencoder approach may become a foundation for interpretability research, helping researchers understand how dangerous behaviors encode within model activations.

• →DACO enables inference-time safety control without retraining frozen multimodal models, reducing computational barriers
• →A curated dictionary of 15,000 multimodal concepts derived from 400,000 stimuli provides granular activation steering capabilities
• →The framework maintains model performance on standard benchmarks while improving safety across multiple MLLM architectures
• →Sparse autoencoders initialized with dictionary concepts automatically annotate semantic meanings, improving interpretability
• →The approach addresses limitations of prompt engineering and response classification by directly intervening on internal representations