Merlin: A Vision Language Foundation Model for 3D Computed Tomography

Area of Science:

• Artificial Intelligence in Medical Imaging
• Radiology and Medical Informatics
• Computer Vision and Natural Language Processing

Background:

• Millions of abdominal CT scans require radiologist interpretation annually in the US.
• Current AI models for medical imaging often limited to 2D data and short reports.
• Need for advanced AI to assist radiologists and extract novel insights from complex scans.

Purpose of the Study:

• Introduce Merlin, a 3D vision-language model (VLM) for abdominal CT interpretation.
• Leverage electronic health records (EHR) and radiology reports for pretraining without manual annotation.
• Enhance automated medical image analysis and extract physiological insights from CT scans.

Main Methods:

• Trained Merlin on a large clinical dataset: 6+ million CT images, 1.8+ million EHR diagnosis codes, and 6+ million radiology report tokens.
• Evaluated Merlin on 6 task types including zero-shot classification, cross-modal retrieval, chronic disease prediction, report generation, and 3D segmentation.
• Performed internal and external validation on diverse CT datasets, including public benchmarks.

Main Results:

• Merlin demonstrated favorable performance compared to existing task-specific baselines across multiple evaluation tasks.
• The model achieved strong results in zero-shot findings and phenotype classification, and cross-modal retrieval.
• Efficient training achieved on a single GPU, indicating potential for democratized AI model development.

Conclusions:

• Merlin represents a significant advancement in 3D VLMs for abdominal CT interpretation.
• The model's ability to integrate EHR data and unstructured reports offers a powerful tool for clinical decision support.
• The computationally efficient training approach facilitates broader adoption of advanced AI in healthcare settings.