A generalist biomedical vision-language model via multi-CLIP knowledge distillation

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Contrastive Language-Image Pretraining (CLIP) models excel in zero-shot and cross-modal tasks using natural images.
• Applying CLIP to biomedicine is hindered by scarce data and diverse imaging types.

Purpose of the Study:

• To develop a generalist biomedical foundation model using multimodal medical knowledge distillation.
• To enhance the capabilities of CLIP models in the biomedical domain.

Main Methods:

• Introduced MMKD-CLIP, integrating knowledge from nine existing biomedical CLIP models.
• Employed a two-stage pipeline: CLIP-style pretraining on 2.9 million medical image-text pairs across 26 modalities, followed by large-scale feature-level distillation.

Main Results:

• MMKD-CLIP demonstrated favorable performance compared to teacher models.
• Achieved robust and cross-domain generalization across 58 datasets and nine modalities.
• Evaluated on tasks including classification, retrieval, visual question answering, survival prediction, and cancer diagnosis.

Conclusions:

• Multimodal medical knowledge distillation is an effective strategy for building powerful biomedical foundation models.
• MMKD-CLIP offers a robust solution for leveraging large-scale biomedical image-text data.
• The model shows significant potential for advancing AI applications in healthcare and medical research.