Development and Validation of a Multimodal Multitask Vision Foundation Model for Generalist Ophthalmic Artificial...

Specialized single-use, single-modality models often have limited or no generalization to new diseases, modalities, and clinical tasks. Foundation models are built for multipurpose use, enabling them to perform tasks even when not specifically pretrained for them, and to adapt to different clinical applications.

We present VisionFM, an artificial intelligence foundation model for ophthalmology pretrained on 3.4 million images from over 500,000 individuals, covering diverse ophthalmic diseases, imaging modalities and devices, and clinical scenarios. Pretrained based on eight modalities, VisionFM was tested for multiple applications, including disease screening and detection, prognosis and prediction, and segmentation of lesions and anatomical structures, on an ophthalmic database comprising 53 public and 12 private datasets. We compared the model against ophthalmologists with varying experience in ophthalmic and systemic disease diagnoses.

VisionFM outperformed baseline deep learning approaches in diagnosing ocular diseases, achieving an average area under the receiver operating characteristic curve (AUROC) of 0.950 (95% confidence interval [CI], 0.941 to 0.959) across eight disease categories and five imaging modalities in internal validation. In external validation, VisionFM achieved an AUROC of 0.945 (95% CI, 0.934 to 0.956) in fundus-based diabetic retinopathy recognition, and an AUROC of 0.974 (95% CI, 0.966 to 0.983) in optical coherence tomography-based age-related macular degeneration recognition. In a comparative study of diagnostic accuracy of 12 ocular diseases from fundus photographs, VisionFM shows diagnostic accuracy close to that of intermediate-level ophthalmologists. Its generalizability extends to new imaging modalities and devices, effectively handling dataset shifts. For example, VisionFM accurately graded diabetic retinopathy with an AUROC of 0.935 (95% CI, 0.902 to 0.964) using an imaging modality it was never exposed to during pretraining. Furthermore, VisionFM is able to predict both glaucoma progression and the presence of intracranial tumors directly from fundus photographs.

VisionFM provides an efficient platform for diagnosis or prediction of multiple diseases using multiple imaging modalities and is scalable to incorporate additional data, modalities, and applications via its open-sourced model weights and codebase. (Funded by the Research Grants Council (RGC) of Hong Kong SAR and others.).