Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis

Researchers propose a method to adapt 2D multimodal large language models for 3D medical imaging analysis, introducing a Text-Guided Hierarchical Mixture of Experts framework that enables task-specific feature extraction. The approach demonstrates improved performance on medical report generation and visual question answering tasks while reusing pre-trained parameters from 2D models.

This research addresses a critical bottleneck in medical AI development: the scarcity of 3D medical imaging data limits training of specialized models from scratch. Rather than building 3D medical MLLMs independently, the researchers leverage transfer learning from well-established 2D models, a pragmatic approach that preserves billions of parameters already optimized for visual understanding. This strategy sidesteps the data limitation problem by repurposing existing knowledge.

The innovation centers on the Text-Guided Hierarchical Mixture of Experts framework, which uses natural language prompts to guide the model toward task-specific processing. This design enables a single model to excel at multiple clinical tasks—medical report generation and visual question answering—without requiring separate specialized systems. The two-stage training strategy differentiates between shared visual features and task-specific patterns, allowing fine-grained adaptation.

The advancement carries meaningful implications for clinical deployment. Hospitals and diagnostic centers currently rely on fragmented AI systems for different imaging analyses, increasing integration complexity and computational overhead. A unified, efficient model capable of handling multiple downstream tasks reduces infrastructure costs and accelerates diagnostic workflows. The reuse of pre-trained parameters also lowers computational requirements compared to training from scratch, making deployment more accessible to resource-constrained healthcare environments.

The acceptance and code release of this work could establish transfer learning as the standard methodology for 3D medical imaging AI, influencing how future research approaches the data scarcity problem. Broader adoption could accelerate AI integration in radiology departments globally, particularly in regions where training medical imaging datasets remain limited.

• →Transfer learning from 2D models enables efficient adaptation to 3D medical imaging without massive new datasets.
• →Task-guided mixture of experts allows a single model to handle multiple clinical applications effectively.
• →The approach reduces computational requirements and infrastructure complexity for medical AI deployment.
• →Results outperform existing 3D medical MLLMs on both report generation and visual question answering benchmarks.
• →Code release could establish transfer learning as standard practice for 3D medical image analysis.