Evaluating Large Language Models in Ophthalmology: Systematic Review.

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Informatics

Background:

• Large language models (LLMs) offer transformative potential in ophthalmic care.
• Current LLM evaluation practices in ophthalmology are fragmented and lack systematic assessment.
• A comprehensive evaluation is needed to identify research gaps and guide clinical integration.

Purpose of the Study:

• To systematically map the current landscape of LLM evaluations in ophthalmology.
• To assess the feasibility of synthesizing LLM performance data for common ophthalmic tasks.

Main Methods:

• A systematic literature search was performed across major databases (PubMed, Web of Science, Embase, IEEE Xplore) up to November 17, 2024.
• 187 studies quantitatively assessing LLMs in ophthalmology were included, with data extracted on LLM type, data modality, subspecialty, task, evaluation dimension, and clinical alignment.
• Descriptive statistics, Fisher exact tests, and an exploratory random-effects meta-analysis were conducted.

Main Results:

• The majority of studies (187) focused on closed-source LLMs (e.g., ChatGPT, Gemini) and text-only evaluations (n=168).
• Open-source LLMs were underrepresented (13.4%), particularly in pure evaluation studies (4.8%).
• Evaluations were skewed towards comprehensive ophthalmology and diagnosis-making tasks, with limited exploration of multimodal data, non-English contexts, or real-world deployment; meta-analysis showed high heterogeneity (I²=94.5%).

Conclusions:

• LLM evaluations in ophthalmology are abundant but highly heterogeneous, limiting performance aggregation.
• Significant gaps exist in evaluating open-source models, multimodal tasks, and real-world applicability.
• Standardized benchmarks and phased clinical validation are essential for safe LLM integration into eye care.