Multimodal Diagnostic Accuracy of GPT-4.o, Claude 3.7, and Gemini 2.5 on Real-World Retina Cases

Purpose: To evaluate the diagnostic accuracy of GPT-4.o, Claude 3.7 Sonnet, and Gemini 2.5 Pro for real-world retinal cases. Methods: Forty retina cases from the University of Iowa's EyeRounds repository were assessed under 2 conditions: (1) full clinical context (textual history, examination, and image descriptions); and (2) image-only (raw clinical images). Outputs were evaluated for diagnostic accuracy, symptom/sign recognition, differential diagnosis, and treatment recommendations. Cochran Q, McNemar test, analysis of variance, and Cohen kappa were used to make statistical comparisons. Results: In full-context cases, GPT-4.o achieved the highest diagnostic accuracy (78.4%), followed by Claude (73.0%) and Gemini (29.7%) (P < .001). In image-only cases, Claude was superior (73.7%), outperforming GPT-4.o (63.2%) and Gemini (31.6%) (P < .001). GPT-4.o was best at identifying signs/symptoms in full-context cases (64.0%), while Claude excelled in image-only cases (63.3%). GPT-4.o (42.9%) and Claude (37.0%) outperformed Gemini (25.9%) on differential diagnosis (P ≤ .011). Claude showed superior treatment accuracy in image-only cases (61.1%), while GPT-4.o was superior in full-context cases (57.3%), and both outperformed Gemini (35.8%). GPT-4.o and Claude showed substantial agreement in image-based diagnoses (κ=0.658), while Gemini showed minimal agreement (κ≤0.196). Conclusions: GPT-4.o and Claude demonstrated strong diagnostic and clinical reasoning in retinal cases, with Claude excelling in image-based analysis and GPT-4.o in text-rich contexts. Gemini's lower performance underscores the importance of careful model selection in clinical applications.