Improving Medical Vision-Language Contrastive Pretraining With Semantics-Aware Triage.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Medical vision-language pretraining (VLP) uses contrastive loss on paired image-report data.
• Current methods treat all unpaired data as negative samples, which can harm representation learning due to inherent similarities in medical data.

Purpose of the Study:

• To develop a more effective contrastive learning strategy for medical VLP.
• To address the limitations of treating all unpaired medical data as negative samples.

Main Methods:

• A novel approach simplifies similarity computation by focusing on inter-report similarity.
• Image-report tuples are categorized into positive, negative, and neutral groups for refined contrastive loss construction.
• The model-agnostic strategy was applied to two state-of-the-art pretraining frameworks.

Main Results:

• Consistent improvements were observed across four downstream tasks: cross-modal retrieval, zero-shot image classification, data-efficient image classification, and image segmentation.
• The proposed strategy demonstrated effectiveness when integrated with existing VLP frameworks.

Conclusions:

• The refined sample categorization significantly enhances contrastive learning in the medical vision-language domain.
• This approach offers a more robust and effective method for medical VLP, leading to better performance on critical clinical applications.