Rethinking masked image modelling for medical image representation.

Area of Science:

• Computer Vision
• Medical Imaging
• Self-Supervised Learning

Background:

• Masked Image Modelling (MIM) excels in computer vision using unannotated data.
• Traditional random masking in MIM may not be optimal for sparse, localized features in medical images.
• Medical images often have accompanying reports that can guide feature identification.

Purpose of the Study:

• To introduce Masked medical Image Modelling (MedIM), a novel approach using radiological reports to guide image masking.
• To enhance semantic representations in medical images by focusing on informative regions.
• To develop report-guided masking strategies for improved self-supervised learning in medical imaging.

Main Methods:

• Proposed MedIM, the first method to use radiological reports for masking in self-supervised learning.
• Introduced knowledge-driven masking (KDM) using Medical Subject Headings (MeSH) for symptom clues.
• Developed sentence-driven masking (SDM) integrating sentence-level report information for targeted masking.
• Reconstructed images using KDM and SDM to create enriched medical image representations.

Main Results:

• MedIM achieved competitive performance across seven downstream tasks, including classification and segmentation.
• Report-guided masking in MedIM significantly outperformed ImageNet pre-training, standard MIM, and medical image-report pre-training.
• Demonstrated the effectiveness of KDM and SDM in promoting stronger semantic representations.

Conclusions:

• MedIM offers a novel and effective approach for self-supervised learning in medical imaging by leveraging radiological reports.
• Report-guided masking is a promising strategy for improving medical image representation and downstream task performance.
• The proposed method provides a strong baseline for future research in medical image self-supervised learning.