Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise

Researchers introduce VisPrompt, a framework that improves prompt learning for vision-language models by injecting visual semantic information to enhance robustness against label noise. The approach keeps pre-trained models frozen while adding minimal trainable parameters, demonstrating superior performance across seven benchmark datasets under both synthetic and real-world noisy conditions.

VisPrompt addresses a critical gap in prompt learning research by tackling the challenge of label noise, a pervasive problem in real-world machine learning deployments where training data contains mislabeled examples. Traditional prompt learning, while parameter-efficient, remains vulnerable to corrupted labels that can degrade model performance. The researchers leverage a fundamental insight: visual information extracted from images is inherently more robust and semantically rich than potentially noisy text labels, providing a more reliable anchor for learning.

The framework's innovation lies in its cross-modal attention mechanism that reversely injects visual semantics into prompt representations, allowing prompt tokens to selectively aggregate image-specific information relevant to individual samples. This design contrasts with naive approaches that would uniformly apply visual guidance regardless of image quality. The introduction of conditional modulation further refines this process by adaptively controlling injection strength per sample, creating a dynamic balance between learned text semantics and visual evidence.

For the machine learning community, this work demonstrates that vision-language models can achieve stronger robustness without fine-tuning expensive pre-trained backbones, reducing computational costs and democratizing robust AI development. The approach's effectiveness across multiple real-world noisy datasets suggests practical applicability to production environments where label quality varies significantly.

The public code release accelerates adoption among researchers and practitioners. Future developments may extend conditional modulation to other multi-modal architectures or explore automated mechanisms for determining injection strength, advancing the broader field of noise-robust learning in vision-language systems.

• →VisPrompt injects visual semantics into prompts via cross-modal attention to anchor learning to reliable instance-level visual evidence rather than noisy labels.
• →Conditional modulation mechanism adaptively controls visual information injection strength per sample, balancing text priors with image evidence dynamically.
• →The framework maintains frozen pre-trained vision-language model backbones while adding minimal trainable parameters for computational efficiency.
• →Extensive experiments on seven benchmarks under synthetic and real-world label noise demonstrate consistent performance improvements over existing baselines.
• →Public code availability enables broader research community adoption of noise-robust prompt learning techniques for practical applications.