Related Experiment Video
Updated: Jun 8, 2025
07:11
Assessing Early Stage Open-Angle Glaucoma in Patients by Isolated-Check Visual Evoked Potential
Published on: May 25, 2020
6.3K
Performance of Chatgpt in ophthalmology exam; human versus AI
Ali Safa Balci1,2, Zeliha Yazar3, Banu Turgut Ozturk4
1Department of Ophthalmology, Beyoglu Eye Training and Research Hospital, University of Health Sciences, 34420, Istanbul, Turkey. alisafabalci@gmail.com.
International Ophthalmology
|November 6, 2024
Summary
ChatGPT achieved a 53.6% success rate on ophthalmology resident exam questions, performing below the average of a third-year resident. Its responses were difficult to understand, with accuracy decreasing as question difficulty increased.
Area of Science:
- Ophthalmology
- Artificial Intelligence in Medicine
- Medical Education
Background:
- The integration of Artificial Intelligence (AI) tools like ChatGPT into medical education is rapidly evolving.
- Evaluating the performance of AI in specialized medical examinations is crucial for understanding its potential and limitations.
Purpose of the Study:
- To assess the accuracy of ChatGPT's responses to questions from the ophthalmology Resident Training Development Exam.
- To compare ChatGPT's performance against ophthalmology residents across different seniority levels.
Main Methods:
- ChatGPT was presented with 75 exam questions covering nine sections and three difficulty levels.
- Responses and explanations were analyzed for accuracy and readability using the Flesch Reading Ease (FRE) score.
- Resident performance was categorized by seniority and compared with ChatGPT's results.
Main Results:
- ChatGPT answered 53.62% of questions correctly, with the highest success in Lens and Cataract (77.77%) and lowest in Pediatric Ophthalmology and Strabismus (0.00%).
- ChatGPT's overall accuracy (53.62%) was lower than the average resident accuracy (50.37%), ranking below a third-year resident.
- Accuracy decreased with increasing question difficulty, and the average FRE score of ChatGPT's responses was low (27.56 ± 12.40), indicating poor readability.
Conclusions:
- ChatGPT demonstrates a moderate success rate on ophthalmology resident exam questions but underperforms compared to residents, particularly those in higher years.
- The readability of ChatGPT's explanations is a significant concern, hindering its utility in medical training.
- Future iterations of AI models may show improved performance as they incorporate more medical knowledge.
