VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

Researchers introduce VLM-RobustBench, a comprehensive benchmark testing vision-language models across 133 corrupted image settings. The study reveals that current VLMs are semantically strong but spatially fragile, with low-severity spatial distortions often causing more performance degradation than visually severe photometric corruptions.

• →VLM-RobustBench evaluates vision-language models across 49 augmentation types and 133 corrupted image settings.
• →Visual severity is a weak predictor of difficulty, with low-severity spatial perturbations often degrading performance more than severe photometric corruptions.
• →Low-severity glass blur reduces MMBench accuracy by 8 percentage points on average across models.
• →Geometric distortions like upsample and elastic transform cause the largest performance drops, reaching up to 34 percentage points.
• →Current vision-language models demonstrate semantic strength but significant spatial fragility under real-world image distortions.