StruXLIP: Enhancing Vision-language Models with Multimodal Structural Cues

StruXLIP is a new fine-tuning paradigm for vision-language models that uses edge maps and structural cues to improve cross-modal retrieval performance. The method augments standard CLIP training with three structure-centric losses to achieve more robust vision-language alignment by maximizing mutual information between multimodal structural representations.

• →StruXLIP introduces a novel approach using edge maps as proxies for visual structure to enhance vision-language model alignment.
• →The method outperforms current competitors on cross-modal retrieval tasks in both general and specialized domains.
• →Three structure-centric losses are used: aligning edge maps with structural text, matching local edge regions to textual chunks, and connecting edge maps to color images.
• →The approach can be integrated into future vision-language models as a plug-and-play boosting recipe.
• →Code and pretrained models are publicly available for researchers and developers.