Evaluating AI Versus Examiner Feedback in Ophthalmology Exit Examinations: A Pilot Study

Area of Science:

• Ophthalmology
• Medical Education
• Artificial Intelligence

Background:

• Preparing for the Royal College of Ophthalmologists Part 2 Oral Examination is crucial for aspiring ophthalmologists.
• Simulated scenarios are vital for honing clinical and communication skills.
• Evaluating feedback quality is key to effective exam preparation.

Purpose of the Study:

• To compare the quality of feedback from generative AI and a Royal College Examiner.
• To assess AI's utility in preparing candidates for ophthalmology oral examinations.
• To identify similarities and differences in feedback content and structure.

Main Methods:

• Utilized GPT-3.5 and GPT-4 to create an interactive web-based platform for simulated oral examinations.
• Employed a standardized patient for five clinical and communication scenarios.
• Thematically analyzed transcripts of AI (Examiner A) and Royal College Examiner (Examiner B) feedback using NVivo software.

Main Results:

• Both AI and human examiners provided feedback on similar themes like empathy, clarity, and clinical reasoning.
• AI feedback was more structured, referencing specific frameworks and offering protocol-driven guidance.
• Human examiner feedback was more practical, context-specific, and insight-driven, based on experience.

Conclusions:

• Generative AI can supplement traditional oral exam preparation by offering accessible, structured feedback.
• AI feedback may establish an early learning foundation for candidates, especially those new to Royal College exams.
• AI tools show potential to enhance the scalability and accessibility of exam preparation resources.