Comparative Accuracy and Safety of Large Language Models in Glaucoma Knowledge Assessment
Related Experiment Video
Updated: Nov 4, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Bora Deniz Argon1, Zeynep Korkut
1Department of Ophthalmology, University of Health Sciences, Prof. Dr. Cemil Taşçıoğlu City Hospital, Şişli, Istanbul, Türkiye.
Prcis:
Across glaucoma MCQs, GPT-5.2, Gemini 3 Pro, and Claude Opus 4.5 were highly accurate, whereas DeepSeek-V3.2 underperformed; potentially harmful wrong answers were uncommon.
Purpose:
To compare the accuracy and potential patient-safety risk of four large language models (LLMs) on glaucoma multiple-choice questions (MCQs).
Methods:
DeepSeek-V3.2, GPT-5.2 (via ChatGPT), Gemini 3 Pro, and Claude Opus 4.5 were tested on two American Academy of Ophthalmology datasets: 129 text-only AAO ONE Network glaucoma self-assessment questions and 32 text-only Basic and Clinical Science Course (BCSC) glaucoma study questions. Each item was queried once per model using a standardized prompt in a fresh session. Accuracy was compared using Cochran's Q and pairwise exact McNemar tests with Holm adjustment. Potentially harmful wrong answers (HW2) were coded using a prespecified rubric.
Results:
In AAO ONE, accuracy was 69.0% for DeepSeek-V3.2, 93.0% for GPT-5.2, 92.2% for Gemini 3 Pro, and 96.1% for Claude Opus 4.5 (Q=65.17, P<0.0001). DeepSeek-V3.2 was significantly less accurate than the other three models; differences among the other models were not significant. In BCSC, accuracy was 65.6%, 93.8%, 84.4%, and 87.5%, respectively (Q=18.00, P=0.0004); GPT-5.2 outperformed DeepSeek-V3.2 after correction. A post hoc item-level screen classified 38/161 questions (23.6%) as directly HW2-eligible; four HW2 events occurred and did not differ significantly across models.
Conclusions:
GPT-5.2, Gemini 3 Pro and Claude Opus 4.5 achieved high accuracy on text-only glaucoma MCQs, whereas DeepSeek-V3.2 underperformed. Harmful errors were uncommon but present, underscoring the need for clinician oversight.