A three-dimensional multi-modal foundation model for optical coherence tomography.

Area of Science:

• Ophthalmology
• Medical Imaging
• Artificial Intelligence

Background:

• Retinal diseases are a major cause of global vision loss.
• Optical coherence tomography (OCT) is crucial for retinal disease diagnosis.
• Existing computational models often underutilize 3D OCT data and struggle to integrate multiple imaging types.

Purpose of the Study:

• To introduce OCTCube-M, a 3D OCT-based multi-modal framework.
• To enable integrated analysis of 3D OCT and 2D en face (EF) images.
• To leverage contrastive learning for combining OCT with fundus autofluorescence and infrared retinal imaging (IR).

Main Methods:

• Developed OCTCube-M, a framework utilizing COEP contrastive learning.
• Created three models: OCTCube (uni-modal), OCTCube-IR (bi-modal), and OCTCube-EF (tri-modal).
• Pre-trained OCTCube on 26,605 3D OCT volumes (1.62M slices); trained OCTCube-IR with OCT/IR pairs; trained OCTCube-EF with OCT/EF images.

Main Results:

• OCTCube achieved state-of-the-art retinal disease prediction with strong generalizability.
• OCTCube-IR demonstrated accurate cross-modality retrieval and joint analysis of OCT and IR images.
• OCTCube-EF excelled in predicting geographic atrophy progression across multi-center clinical trials.

Conclusions:

• OCTCube-M is a powerful 3D multi-modal foundation model for retinal imaging analysis.
• The framework significantly advances cross-site, cross-device, and cross-modality disease prediction.
• OCTCube-M offers substantial utility in geographic atrophy clinical trials and systemic disease prediction.