MG-3D: Multi-grained knowledge-enhanced vision-language pre-training for 3D medical image analysis.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Foundation Models

Background:

• Current AI for 3D medical image analysis struggles with generalization and requires costly task-specific annotations.
• Foundation models (FMs) offer potential for broad clinical applications with less annotation.
• Radiology reports, rich in semantics, are underutilized in pre-training FMs for 3D medical imaging.

Purpose of the Study:

• To develop a vision-language pre-training (VLP) method, MG-3D, to effectively utilize large-scale 3D medical image and radiology report data.
• To enhance the understanding of multi-grained semantics within and across 3D medical images and their associated reports.
• To improve the generalization and reduce annotation dependency of AI models in 3D medical image analysis.

Main Methods:

• MG-3D employs cross-modal global alignment and modality-guided local reconstruction for intra-patient semantic correspondence.
• Inter-patient visual semantics are correlated using fine-grained report correlations and contrastive learning.
• Pre-training utilized a large-scale dataset of 47.1K 3D Computed Tomography (CT) scans and reports.

Main Results:

• MG-3D demonstrated superior transferability, scalability, and generalization across ten diverse clinical tasks.
• The method showed efficacy on both internal and external validation datasets, including 3D CT and MRI.
• Evaluations confirmed improved performance in 3D medical image analysis tasks compared to existing methods.

Conclusions:

• MG-3D effectively harnesses the semantic richness of radiology reports for pre-training foundation models in 3D medical imaging.
• The proposed approach addresses key limitations in current AI development for medical image analysis, paving the way for more robust and adaptable AI tools.
• The study highlights the potential of large-scale 3D volume-report data for advancing AI in diagnostics, prognostics, and therapeutics.