CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

Researchers introduce CheXpercept, a benchmark dataset for evaluating vision-language models on chest X-ray analysis that goes beyond simple disease classification to test clinical-grade lesion perception. Testing 14 VLMs reveals that models perform adequately only at basic detection levels, with accuracy declining sharply on more complex visual tasks, and medical-specific models show no meaningful advantage over general models.

CheXpercept addresses a critical gap in AI healthcare evaluation by establishing rigorous benchmarks for clinical AI reliability. Current VLM assessments focus narrowly on whether models can identify disease presence or absence, failing to validate whether they perceive lesions with the precision required for clinical deployment. This benchmark mirrors actual radiologist workflows through three escalating difficulty levels: coarse detection, fine-grained contour analysis, and semantic attribute extraction. The dataset construction involved semi-automated generation paired with expert review, ensuring clinical validity at scale across 2,100 chest X-rays and 10,400 QA items covering seven critical lesion types.

The benchmark results expose significant limitations in current AI development. While models achieve reasonable performance on basic detection tasks, accuracy degrades substantially when finer visual discrimination becomes necessary. More troubling is that medical VLMs—specifically trained for healthcare applications—demonstrate almost no perceptual advantage over general-purpose models. This systemic flaw suggests domain adaptation strategies in medical AI remain immature and that specialized training may not meaningfully improve clinical reliability.

For the AI healthcare industry, these findings signal that current VLMs lack the perceptual sophistication required for clinical deployment in radiology. Healthcare organizations considering VLM adoption must recognize that coarse-level accuracy masks poor performance on clinically critical fine-grained tasks. The public release of CheXpercept and associated code enables standardized evaluation, establishing a foundation for measuring real progress in medical AI perception. Developers face pressure to move beyond architecture improvements toward fundamental advances in visual reasoning that match radiologist expertise.

• →Vision-language models show adequate performance only at basic lesion detection, with accuracy dropping significantly on finer visual tasks.
• →Medical-specialized VLMs provide almost no advantage over general-purpose models, revealing ineffective domain adaptation strategies.
• →CheXpercept benchmark mirrors radiologist cognitive workflows across three complexity levels to ensure clinical-grade evaluation standards.
• →Current VLM architectures lack the perceptual sophistication required for reliable clinical deployment in radiology.
• →Public dataset release enables standardized benchmarking and establishes measurable goals for advancing medical AI perception capabilities.