ES-Merging: Biological MLLM Merging via Embedding Space Signals

Researchers propose ES-Merging, a new framework for combining specialized biological multimodal large language models (MLLMs) by using embedding space signals rather than traditional parameter-based methods. The approach estimates merging coefficients at both layer-wise and element-wise granularities, outperforming existing merging techniques and even task-specific fine-tuned models on cross-modal scientific problems.

• →New ES-Merging framework addresses limitations of existing biological MLLMs that are restricted to single modalities.
• →Method uses embedding space signals instead of input-agnostic parameter heuristics for more effective model merging.
• →Approach estimates coefficients at both coarse-grained (layer-wise) and fine-grained (element-wise) levels for robust performance.
• →Experiments show the method outperforms existing merging techniques and surpasses task-specific fine-tuned models.
• →Research enables better cross-modal scientific discovery by unifying different modality-specialized models.