MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

Researchers introduced MMBU, the largest biomedical vision-language benchmark covering 35 medical imaging modalities with structured metadata. Testing 15 open-weight and 2 frontier VLMs revealed that while medical adaptation helps some models, high reported accuracy on existing benchmarks masks significant deficiencies in visual perception and domain generalization.

The introduction of MMBU addresses a critical gap in evaluating vision-language models for biomedical applications. As VLMs increasingly integrate into clinical workflows—from radiology to pathology—rigorous benchmarking becomes essential for assessing their real-world reliability. The benchmark's scale and diversity represent a substantial advance; with 35 submodalities and rich structured metadata, MMBU enables systematic evaluation across biological scales, clinical contexts, and imaging types that previous benchmarks couldn't adequately cover.

This research emerges as VLMs gain momentum in medical AI adoption. Despite impressive performance metrics on narrow tasks, these models often struggle with subtle visual features critical to diagnosis. The benchmark's dual focus on open and closed task versions—including ungrounded classification, grounded classification, and object detection—provides comprehensive insight into model capabilities and limitations. The finding that medical adaptation yields measurable but inconsistent gains suggests current approaches to domain-specific fine-tuning remain incomplete.

For the AI industry, MMBU establishes a more rigorous evaluation standard that could reshape how biomedical VLMs are developed and deployed. Medical institutions and technology companies relying on these models must now contend with evidence that reported benchmarks may overstate practical performance. This underscores the gap between laboratory metrics and clinical utility, particularly for high-stakes applications where diagnostic accuracy directly impacts patient outcomes.

The benchmark's public release likely catalyzes further refinement of biomedical VLMs. Developers will leverage MMBU to identify and address perception gaps, while researchers can use it to validate novel architectures and training methods. Ongoing evaluation across diverse modalities will highlight which model families best generalize to real clinical scenarios.

• →MMBU is the largest biomedical vision-language benchmark to date, covering 35 medical imaging submodalities with comprehensive structured metadata.
• →Testing showed that reported high accuracy on existing benchmarks often masks significant deficiencies in visual perception and domain generalization.
• →Medical adaptation provides measurable but inconsistent performance gains across different VLM architectures.
• →The benchmark includes both open and closed task variants enabling systematic evaluation of model performance across scales, settings, and imaging types.
• →The research establishes more rigorous evaluation standards for biomedical AI, highlighting gaps between laboratory metrics and real-world clinical utility.