Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Education

Background:

• Artificial intelligence (AI) chatbots like ChatGPT have societal implications.
• AI-driven medical training curricula are emerging.
• The performance of AI chatbots in ophthalmology has not been evaluated.

Purpose of the Study:

• To evaluate ChatGPT's accuracy in answering ophthalmology board certification practice questions.

Main Methods:

• A cross-sectional study analyzed 125 text-based multiple-choice questions from OphthoQuestions.
• ChatGPT's responses were assessed for correctness and compared to board certification standards.
• Performance was analyzed in January and February 2023, including stand-alone questions.

Main Results:

• ChatGPT answered 46% of questions correctly in January 2023, improving to 58% in February 2023.
• Performance varied by specialty, with 79% in general medicine but 0% in retina.
• ChatGPT correctly identified the most common trainee answer 44% of the time.

Conclusions:

• ChatGPT answered less than half of ophthalmology board certification questions correctly.
• Current AI performance is insufficient to significantly aid in board certification preparation.
• Medical professionals should note AI advancements but recognize current limitations.