Contrastive Cross-Modal Pre-Training: A General Strategy for Small Sample Medical Imaging

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Machine Learning

Background:

• Training neural networks for medical imaging tasks is hindered by the scarcity of manually labeled examples.
• Medical imaging reports contain valuable expert interpretations but are unstructured.
• There is a need for methods that leverage existing clinical data to improve AI model training.

Purpose of the Study:

• To develop a method for training neural networks for medical image interpretation using unstructured text reports as weak supervision.
• To reduce the dependency on large, manually labeled datasets for medical AI development.
• To enhance the performance of neural networks in medical image analysis without additional labeling efforts.

Main Methods:

• Utilized an image-text matching task to train a neural network feature extractor.
• Employed transfer learning to fine-tune the feature extractor on a small labeled dataset.
• Developed a system that interprets medical imagery without requiring text reports during inference.

Main Results:

• Achieved consistent performance improvements across three medical image classification tasks.
• Demonstrated a significant reduction in the requirement for labeled data, ranging from 67% to 98%.
• The trained neural network effectively interprets medical imagery.

Conclusions:

• Leveraging unstructured textual imaging reports as weak supervision is an effective strategy for training medical AI.
• This approach substantially decreases the need for manual data labeling in medical imaging AI.
• The proposed method offers a practical solution for improving AI-driven medical image interpretation in clinical settings.