Foundation models in ophthalmology: opportunities and challenges

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Imaging

Background:

• The advent of foundation models, such as RETFound, marks a significant step in ophthalmology.
• Advancements in large language models (LLMs) like GPT-4 and Gemini are being adapted for medical applications.
• Generalizable medical artificial intelligence (GMAI) shows potential for adapting to new clinical tasks.

Purpose of the Study:

• To review the opportunities and challenges in advancing foundation models in ophthalmology.
• To explore the potential of large language models and multimodal models in ophthalmology.
• To identify current limitations and future directions for AI in eye care.

Main Methods:

• Review of recent developments in foundation models and large language models in ophthalmology.
• Analysis of performance metrics for ophthalmology-specific AI models.
• Evaluation of the capabilities of large multimodal models (LMMs) compared to LLMs.

Main Results:

• RETFound demonstrates superior performance over traditional deep learning models, even with limited fine-tuning data.
• Specialized LLMs (Med-Gemini, Medprompt GPT-4) outperform general models in ophthalmology tasks.
• A notable gap exists in ophthalmology-specific multimodal models due to high computational costs and data scarcity.

Conclusions:

• Foundation models offer significant opportunities in ophthalmology, but high-quality, standardized datasets are crucial for training and specialization.
• While large language and vision models have advanced, large multimodal models present the greatest potential for mimicking clinical expertise.
• Addressing data limitations and computational resource requirements is key to unlocking the full potential of AI in ophthalmology.