A multimodal visual-language foundation model for computational ophthalmology.

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Imaging

Background:

• Early detection of eye diseases is crucial for preventing vision loss.
• Current AI models often use single data types and struggle with rare conditions.
• There's a need for AI that integrates multi-view information for comprehensive eye disease diagnosis.

Purpose of the Study:

• To introduce EyeCLIP, a multimodal visual-language foundation model for ophthalmology.
• To address limitations of single-modality AI in detecting rare and common eye diseases.
• To leverage a large dataset and novel pretraining for improved diagnostic capabilities.

Main Methods:

• Trained EyeCLIP on 2.77 million ophthalmology images across 11 modalities with clinical text.
• Employed a pretraining strategy combining self-supervised reconstruction, multimodal contrastive learning, and image-text contrastive learning.
• Evaluated performance on 14 benchmark datasets for various ophthalmic AI tasks.

Main Results:

• EyeCLIP demonstrated robust performance in disease classification, visual question answering, and cross-modal retrieval.
• Achieved strong few-shot and zero-shot learning capabilities, crucial for long-tail disease distributions.
• Showcased effectiveness in detecting both ocular and systemic diseases from medical images.

Conclusions:

• EyeCLIP represents a significant advancement in multimodal AI for ophthalmology.
• The model's ability to handle diverse data and rare diseases offers potential for real-world clinical applications.
• EyeCLIP can bridge gaps in current diagnostic tools, improving patient outcomes through early detection.