Dual-Modality Multi-Stage Adversarial Safety Training: Robustifying Multimodal Web Agents Against Cross-Modal Attacks

Researchers developed DMAST, a new training framework that protects multimodal web agents from cross-modal attacks where adversaries inject malicious content into webpages to deceive both visual and text processing channels. The method uses adversarial training through a three-stage pipeline and significantly outperforms existing defenses while doubling task completion efficiency.

• →Multimodal web agents are vulnerable to cross-modal attacks that simultaneously corrupt both visual and text observation channels.
• →Attacks with visual components significantly outperform text-only injections, exposing gaps in current VLM safety training.
• →DMAST framework uses a three-stage training pipeline including imitation learning, supervised fine-tuning, and adversarial reinforcement learning.
• →The approach formulates agent-attacker interaction as a two-player zero-sum Markov game for robust training.
• →DMAST doubles task completion efficiency while substantially reducing adversarial risks on out-of-distribution tasks.