Multi-modal representation learning in retinal imaging using self-supervised learning for enhanced clinical...

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Imaging

Background:

• Self-supervised learning (SSL) is crucial for generalizable AI in medical imaging.
• Contrastive representation learning on multi-modal data yields transferable features.
• Ophthalmology offers abundant multi-modal retinal imaging data (2D fundus, 3D OCT).

Purpose of the Study:

• Introduce a novel multi-modal contrastive learning pipeline for joint retinal imaging representation.
• Evaluate the transferability and generalizability of learned representations for downstream tasks.
• Assess the impact of using lower-cost fundus imaging instead of OCT.

Main Methods:

• Developed a multi-modal contrastive learning framework for 2D fundus and 3D OCT scans.
• Performed self-supervised pre-training on 153,306 retinal scan pairs.
• Validated the framework on three independent external datasets for clinical prediction tasks.

Main Results:

• The pre-training framework generated effective retrieval systems and encoders.
• Learned representations generalized well across various downstream tasks.
• Replacing OCT with fundus imaging maintained significant predictive power.

Conclusions:

• Multi-modal contrastive learning effectively creates joint representations for retinal imaging.
• The approach yields transferable features for diverse ophthalmology applications.
• Fundus imaging alone can suffice for certain predictive tasks, offering a cost-effective alternative.