Radiology Reports Improve Visual Representations Learned from Radiographs.

Area of Science:

• Biomedical Informatics
• Computer Vision
• Machine Learning

Background:

• Human perception integrates multi-modal stimuli (acoustic, verbal, visual).
• Multi-modal learning frameworks are effective for natural datasets but less explored in biomedicine.
• Radiology data (images and reports) offers abundant unstructured multi-modal information.

Purpose of the Study:

• To adapt and evaluate multi-modal and self-supervised learning frameworks for biomedical data, specifically chest radiograph classification.
• To compare the effectiveness of multi-modal learning, self-supervised learning, and joint learning strategies for visual representation in this domain.

Main Methods:

• Leveraging unstructured multi-modal data from radiology images and reports.
• Implementing and comparing multi-modal learning, self-supervised learning, and a combination (joint learning).
• Evaluating performance on downstream chest radiograph classification tasks, particularly with limited labeled data (1% and 10%).

Main Results:

• Joint learning with multi-modal and self-supervised models outperformed self-supervised learning in limited labeled data settings (1% and 10%).
• Joint learning performed comparably to multi-modal learning under these conditions.
• Multi-modal learning demonstrated greater robustness on out-of-distribution datasets.

Conclusions:

• Joint learning strategies effectively enhance visual representations for chest radiograph classification, especially when labeled data is scarce.
• Multi-modal learning offers robustness, making it a valuable approach for biomedical applications.
• The study provides insights into optimizing learning strategies for medical imaging analysis.