Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models

Researchers propose a new training method called pseudo contrastive learning to improve diagram comprehension in multimodal AI models like CLIP. The approach uses synthetic diagram samples to help models better understand fine-grained structural differences in diagrams, showing significant improvements in flowchart understanding tasks.

• →Current multimodal models like CLIP struggle with diagram comprehension due to limited sensitivity to fine-grained structural variations.
• →The new pseudo contrastive learning method generates synthetic diagrams using randomly picked text elements to create training samples.
• →The approach enhances diagram understanding without requiring modification of original training data.
• →Empirical tests on flowchart datasets show substantial improvements over standard CLIP training methods.
• →The research contributes to advancing domain-specific training strategies for vision-language models.