Multimodal masked siamese network improves chest X-ray representation learning.

Area of Science:

• Artificial Intelligence in Medical Imaging
• Machine Learning for Healthcare
• Radiology Informatics

Background:

• Current self-supervised learning for medical images often overlooks valuable Electronic Health Records (EHR) data.
• Integrating diverse patient and scan information can potentially improve model performance.

Purpose of the Study:

• To develop a multimodal pretraining strategy for chest radiographs incorporating EHR data.
• To enhance the quality of chest radiograph representations using a Masked Siamese Network (MSN).

Main Methods:

• Proposed a multimodal Masked Siamese Network (MSN) integrating EHR data (demographics, scan metadata, inpatient stay) with chest radiographs.
• Evaluated the approach on MIMIC-CXR, CheXpert, and NIH-14 datasets using ViT-Tiny and ViT-Small backbones.
• Assessed representation quality via linear evaluation and Area Under the Receiver Operating Characteristic Curve (AUROC).

Main Results:

• The multimodal MSN significantly improved representation quality over vanilla MSN and state-of-the-art baselines.
• Achieved 2% AUROC improvement over vanilla MSN and 5-8% over other baselines.
• Demographic features demonstrated the most substantial performance enhancement.

Conclusions:

• EHR-enhanced self-supervised pretraining offers a promising avenue for improving medical imaging analysis.
• This multimodal strategy can advance representation learning for various medical imaging modalities.
• Future research can explore this approach for neuroimaging, ophthalmic imaging, and sonar imaging.