CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuning.

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Computer Vision

Background:

• Contrastive Language-Image Pre-training (CLIP) shows promise in self-supervised learning but requires significant computational resources and large datasets, limiting its medical application.
• Current CLIP methods often rely on manually derived prompts from image labels, potentially missing nuanced clinical information.
• Medical datasets are often smaller and less accessible, posing challenges for resource-intensive models.

Purpose of the Study:

• To introduce an efficient language-image contrastive learning method (CLEFT) for medical applications.
• To develop a strategy for learning context-based prompts that better utilize clinical data.
• To improve the performance and efficiency of self-supervised representation learning in medical imaging.

Main Methods:

• Developed CLEFT, a novel contrastive learning framework integrating efficient large language models and prompt fine-tuning.
• Implemented an efficient strategy for learning context-based prompts from clinical diagnostic data.
• Leveraged extensive pre-trained language and visual models for enhanced representation learning.

Main Results:

• Achieved state-of-the-art performance on multiple chest X-ray and mammography datasets.
• Demonstrated significant reductions in trainable model size (39%) and language model parameters (to 4% of BERT encoder).
• Showcased the effectiveness of context-based prompts in bridging the gap between clinical data and simple labels.

Conclusions:

• CLEFT offers a parameter-efficient and high-performing alternative to existing CLIP-like methods for medical imaging.
• The proposed prompt learning strategy enhances the utility of clinical data for self-supervised learning.
• CLEFT's efficiency makes it suitable for medical applications with limited data and computational resources.