Assessing a large language model for glaucoma knowledge: ChatGPT-5 versus residents

Area of Science:

• Ophthalmology
• Artificial Intelligence in Medicine
• Medical Education Technology

Background:

• Large language models (LLMs) are increasingly integrated into various professional fields.
• Assessing the performance of LLMs against human experts is crucial for understanding their capabilities and limitations.
• Glaucoma education and assessment tools are vital for training ophthalmologists.

Purpose of the Study:

• To compare the performance of a contemporary LLM, ChatGPT-5, against ophthalmology residents.
• To evaluate the accuracy of ChatGPT-5 on standardized glaucoma multiple-choice questions.
• To identify areas where LLMs excel or fall short compared to resident knowledge.

Main Methods:

• A cross-sectional study utilized 189 text-only glaucoma multiple-choice questions from the Cybersight question bank.
• ChatGPT-5 was tested under standardized, text-only conditions.
• Six ophthalmology residents (PGY1-3) answered the same questions under supervision, with accuracy compared using McNemar's exact test.

Main Results:

• ChatGPT-5 achieved 86.8% accuracy, significantly outperforming residents (62.9% overall accuracy).
• ChatGPT-5 outperformed all residents in head-to-head comparisons (ORs 1.84-13.15, p≤0.023).
• Residents were more successful on 9.0% of items that ChatGPT-5 answered incorrectly.

Conclusions:

• ChatGPT-5 demonstrates potential as a valuable tool for ophthalmology education and assessment.
• Current LLM performance is limited by text-only data and specific question banks.
• Further research with multimodal data and larger cohorts is needed to confirm generalizability and clinical utility.