Generalist foundation models from a multimodal dataset for 3D computed tomography.

Area of Science:

• Medical Imaging AI
• Computer Vision
• Radiology

Background:

• Advancements in 3D medical imaging AI are hindered by a lack of comprehensive datasets.
• Existing datasets often lack paired 3D image and textual report data.

Purpose of the Study:

• To introduce CT-RATE, a novel public dataset of 3D chest CT scans with corresponding radiology reports.
• To develop and evaluate CT-CLIP, a contrastive language-image pretraining framework for 3D medical imaging.
• To create CT-CHAT, a vision-language chat model for 3D chest CT analysis.

Main Methods:

• Compiled CT-RATE dataset with 25,692 3D chest CT scans and 21,304 patient reports.
• Developed CT-CLIP, a CT-focused contrastive language-image pretraining framework.
• Created CT-CHAT by combining CT-CLIP's vision encoder with a large language model, fine-tuned on 2.7 million Q&A pairs.

Main Results:

• CT-CLIP demonstrated superior performance in multi-abnormality detection and case retrieval compared to state-of-the-art models.
• CT-CHAT showcased the potential of specialized vision-language models for 3D medical imaging.
• The open-source release facilitates further research and development in medical AI.

Conclusions:

• The release of CT-RATE, CT-CLIP, and CT-CHAT addresses key challenges in 3D medical imaging AI.
• These resources pave the way for innovations in AI-driven diagnostics and patient care.
• Specialized models are crucial for advancing 3D medical imaging analysis.