Merlin: a computed tomography vision-language foundation model and dataset

Area of Science:

• Artificial Intelligence in Medical Imaging
• Radiology and Medical Diagnostics
• Computer Vision and Natural Language Processing

Background:

• Increasing volume of abdominal computed tomography (CT) scans necessitates automated analysis tools due to radiologist shortages.
• Existing vision-language models (VLMs) for medical imaging are often limited to 2D data and concise reports, hindering comprehensive abdominal CT interpretation.

Purpose of the Study:

• To introduce Merlin, a 3D VLM designed for in-depth abdominal CT interpretation, overcoming limitations of previous 2D-based approaches.
• To develop a VLM capable of learning from volumetric CT scans, electronic health records, and radiology reports without manual annotations.

Main Methods:

• Developed Merlin, a 3D VLM, utilizing a multistage pretraining framework on a large clinical dataset (>6 million images, >1.8 million diagnosis codes, >6 million report tokens).
• Evaluated Merlin on 6 task types and 752 individual tasks, including zero-shot classification, cross-modal retrieval, chronic disease prediction, report generation, and 3D semantic segmentation.
• Validated Merlin through extensive internal (5,137 scans) and external testing (44,098 scans) across multiple institutions and public datasets.

Main Results:

• Merlin demonstrated high generalization across institutions and anatomies, outperforming existing 2D VLMs, CT foundation models, and off-the-shelf radiology models.
• Achieved strong performance in zero-shot classification of 30 findings and 692 phenotypes, and effective cross-modal retrieval.
• Showcased proficiency in adapted tasks, including 5-year chronic disease prediction for 6 diseases, radiology report generation, and 3D semantic segmentation of 20 organs.

Conclusions:

• Merlin represents a significant advancement in automated abdominal CT interpretation, assisting radiologists and mitigating workload.
• The 3D VLM holds potential for biomarker discovery and disease risk stratification, adding value beyond diagnostic tasks.
• The release of trained models, code, and a dataset (25,494 pairs) facilitates further research and development in medical AI.