A large comprehensive comparison of large language models on ophthalmology board exams

A comprehensive performance evaluation of seven state-of-the-art large language models (LLMs) compared to human resident benchmarks on ophthalmology board exam questions. Furthermore, this study aims to define current model strengths and limitations while validating an assessment tool for clinicians to evaluate model-generated outputs.

Seven LLMs were assessed: ChatGPT-5, ChatGPT-4, Gemini 2.5 Pro, Gemini 2.5 Flash, Claude Sonnet 4.5, Grok-4-Fast-Reasoning, and Perplexity Sonar Pro. A dataset of 1,037 Israeli ophthalmology board questions (2020-2025) was manually categorized by question type (logical vs. informative), image modality, and 12 subspecialties. Models were evaluated for accuracy compared with resident performance, response latency, question difficulty, and self-assessed confidence.

Performance varied substantially across models. Gemini 2.5 Pro achieved the highest accuracy, followed by ChatGPT-5, both outperforming residents. Accuracy declined significantly with increasing question difficulty (Very Hard vs. Easy: aOR 0.28, 95% CI,0.16-0.46; aP < 0.001), mirroring resident trends. Questions containing images were significantly more challenging for all models. Logical reasoning questions were significantly more challenging for LLMs than informative ones (aOR 0.66, aP = 0.019). Confidence calibration varied widely, with ChatGPT-5 showing superior discrimination (AUC = 0.827).

LLMs' performance on ophthalmology board questions varied across models. Gemini 2.5 Pro achieved the highest performance, surpassing the mean resident score. However, the majority of LLMs exhibit significant error rates and poor calibration of self-assessed confidence. To safely integrate LLMs into the clinical workflow, continuous evaluation is essential to guide ophthalmologists on LLMs' characteristics and clinical limitations.