DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents

Researchers introduce DeepTumorVQA, a comprehensive benchmark for evaluating medical AI vision-language models on 3D CT tumor analysis through 476K hierarchical questions across four diagnostic stages. The study reveals that measurement accuracy is the critical bottleneck in medical AI reasoning, and that tool-augmented agents significantly outperform models working without external resources.

DeepTumorVQA addresses a fundamental gap in medical AI evaluation by moving beyond single accuracy metrics to decompose complex diagnostic reasoning into measurable stages. Traditional VQA benchmarks collapse capabilities into aggregate scores, making it difficult to identify where models systematically fail in clinical contexts. This hierarchical approach mirrors actual diagnostic workflows—recognition of tumors, precise measurement, visual pattern analysis, and medical reasoning—enabling targeted improvement of AI systems in healthcare.

The benchmark's scale and sophistication reflect growing recognition that medical AI requires domain-specific evaluation frameworks. With 9,262 3D CT volumes and 42 clinical subtypes, DeepTumorVQA provides statistically robust datasets for rigorous assessment. The finding that quantitative measurement represents the primary bottleneck has immediate implications: improving measurement reliability becomes a priority for developers building clinical decision support systems, and this constraint may persist across different model architectures unless explicitly addressed.

Tool-augmentation emerges as a pragmatic solution to measurement limitations, with agent-based approaches substantially improving performance when external segmentation and measurement tools are available. This validates the architectural approach of combining foundation models with specialized clinical software—a hybrid strategy many healthcare AI companies are already pursuing. The research suggests that future medical VLMs should be designed with tool integration as a core capability rather than an afterthought.

Looking forward, the stage-wise framework and released code/data will likely influence how researchers benchmark medical AI systems, potentially shifting industry standards toward more granular evaluation. Organizations developing clinical AI should monitor these emerging evaluation practices, as regulatory bodies and healthcare institutions may adopt similar hierarchical assessment approaches when validating AI systems for diagnostic support.

• →DeepTumorVQA's hierarchical four-stage framework reveals measurement accuracy as the primary bottleneck limiting medical VLM performance in tumor diagnosis
• →Tool-augmented agents substantially mitigate reasoning failures by integrating external segmentation and measurement tools into the reasoning pipeline
• →Existing medical VQA benchmarks obscure model failure modes by collapsing diagnostic reasoning into single accuracy scores, limiting interpretability and targeted improvement
• →The 476K question dataset across 42 clinical subtypes provides sufficient scale to enable robust evaluation of diverse medical AI architectures and approaches
• →Stage-wise ground-truth traces from the benchmark can supervise agent training and reduce both tool-use and medical reasoning errors in production systems