Related Experiment Video
Updated: Sep 14, 2025
Assessing Early Stage Open-Angle Glaucoma in Patients by Isolated-Check Visual Evoked Potential
Published on: May 25, 2020
Evaluating the Use of ChatGPT 3.5 and Bard as Self-Assessment Tools for Short Answer Questions in Undergraduate
Abhijeet M Khake1, Suvarna Gokhale2, Pradeep Dindore3
1Department of Ophthalmology, Pacific Medical College and Hospital, Udaipur, IND.
Objective:
This study aimed to evaluate the efficacy of ChatGPT 3.5 and Google Bard as tools for self-assessment of short answer questions (SAQs) in ophthalmology for undergraduate medical students.
Methodology:
A total of 261 SAQs were randomly selected from previous university examination papers and publicly available ophthalmology question banks. The questions were classified according to the competency-based medical education (CBME) curriculum of the National Medical Commission (NMC) of India into three categories: short note task-oriented questions (SNTO, n = 169), short note reasoning questions (SNRQ, n = 15), and applied aspect SAQs (SN Applied, n = 77). Image-based questions were excluded. Three ophthalmologists collaboratively developed model answers for each question. The same questions were then submitted to ChatGPT 3.5 and Google Bard. The AI-generated responses were independently evaluated by three ophthalmologists using a 3-point scale based on correct diagnosis, accuracy of content, and relevance. The scores were compiled, and the data were analyzed to compare the overall and category-wise performance of the two AI tools.
Results:
Out of a total possible score of 783 (261 questions × 3 points), ChatGPT 3.5 scored 696 (88.8%), while Bard scored 685 (87.5%). Although the overall performance difference was insignificant, ChatGPT 3.5 performed significantly better in the SNTO category. However, both AI tools produced poor-quality or inadequate answers for a subset of questions: 50 (19%) by ChatGPT 3.5 and 44 (16.8%) by Bard. Some responses lacked essential information, even for high-yield topics.
Conclusion:
ChatGPT 3.5 and Bard can generate accurate and relevant responses to ophthalmology SAQs in most cases. ChatGPT 3.5 demonstrated slightly better performance, particularly for task-oriented questions, suggesting it may be a more effective tool for undergraduate students' self-assessment. However, due to a notable error rate (~20%), AI-generated responses should not be used in isolation and must be cross-referenced with standard textbooks. These tools best suit rapid information retrieval during the early study phases.
