A three-dimensional multi-modal foundation model for optical coherence tomography

Vision loss caused by retinal diseases remains a leading global cause of disability. Optical coherence tomography (OCT) is an imaging technique that is used for diagnosing retinal diseases. Computational models can use OCT images for various diagnostic and prognostic tasks, but most existing approaches fail to fully leverage the rich three-dimensional (3D) structure of OCT data and lack the capability to integrate other retinal imaging modalities into the analysis. Here, to address these limitations, we present OCTCube-M, a 3D OCT-based multi-modal framework designed for the integrated analysis of 3D OCT and 2D en face (EF) images. OCTCube-M exploits COEP, an effective multi-modal contrastive learning method, to integrate OCT with other retinal imaging modalities, such as fundus autofluorescence imaging and infrared retinal imaging (IR). Using the OCTCube-M framework, we developed three models: OCTCube (uni-modal), OCTCube-IR (bi-modal) and OCTCube-EF (tri-modal). OCTCube, a 3D foundation model pre-trained on 26,605 3D OCT volumes comprising 1.62 million 2D OCT slices, achieved state-of-the-art performance in predicting 8 retinal diseases while demonstrating robust generalizability across cohorts, devices and modalities. OCTCube-IR extends OCTCube by incorporating 26,685 pairs of OCT and IR images, enabling accurate cross-modality retrieval and joint analysis of these two modalities. OCTCube-EF, trained on over 4 million 2D OCT slices and 400 thousand EF retinal images, excels in predicting the growth rate of geographic atrophy across datasets collected from 6 multi-centre clinical trials across 23 countries. Collectively, OCTCube-M is a 3D multi-modal foundation model framework for integrating OCT and other retinal imaging modalities. It demonstrated substantial advancements in cross-site, cross-device, cross-modality and systemic disease prediction, while offering substantial utility in geographic atrophy clinical trials.