Securing the Floor and Raising the Ceiling: A Merging-based Paradigm for Multi-modal Search Agents

Researchers propose a training-free paradigm for empowering Vision-Language Models with multi-modal search capabilities through cross-modal model merging. The approach uses Optimal Brain Merging (OBM) to combine text-based search agents with base VLMs without requiring expensive supervised training or reinforcement learning.

• →New training-free method enables VLMs to perform multi-modal search without expensive supervised trajectories or reinforcement learning.
• →Optimal Brain Merging (OBM) algorithm identifies critical parameters to reduce interference during cross-modal model integration.
• →Model merging approach provides reasonable zero-shot performance while serving as effective warm-start strategy for faster convergence.
• →Method addresses cold-start problems and training instability issues common in existing multi-modal search agents.
• →Experiments on InfoSeek and MMSearch benchmarks show superior search rates and higher peak accuracy compared to standard VLM initialization.