MedCLIP: Contrastive Learning from Unpaired Medical Images and Text

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computer Vision

Background:

• Contrastive learning methods like CLIP excel at general vision-text tasks.
• Medical datasets are smaller and prone to false negatives in contrastive learning.
• Existing methods struggle with the unique challenges of medical multimodal data.

Purpose of the Study:

• To develop a cost-effective and scalable multimodal contrastive learning framework for medical imaging.
• To address the issue of false negatives in medical vision-text contrastive learning.
• To improve the performance of medical image-text representation learning for downstream tasks.

Main Methods:

• Decoupling images and texts to exponentially increase usable training data.
• Implementing a semantic matching loss leveraging medical knowledge to mitigate false negatives.
• Utilizing a novel framework named MedCLIP for pre-training.

Main Results:

• MedCLIP significantly outperforms state-of-the-art methods in zero-shot prediction, supervised classification, and image-text retrieval.
• Achieved superior performance with only 20K pre-training samples compared to a baseline using ≈200K data.
• Demonstrated the effectiveness of decoupling and semantic matching loss in medical multimodal learning.

Conclusions:

• MedCLIP offers a simple yet highly effective framework for medical vision-text contrastive learning.
• The proposed methods successfully address data scarcity and false negative issues in the medical domain.
• MedCLIP sets a new benchmark for pre-training medical multimodal representations with limited data.