Performance of ChatGPT on optometry and vision science exam questions

Area of Science:

• Optometry and Vision Science
• Artificial Intelligence in Education

Background:

• Large Language Models (LLMs) offer new educational tools but exhibit performance variability across academic domains.
• Assessing LLM capabilities in specialized fields like optometry is crucial for understanding their potential and limitations.

Purpose of the Study:

• To evaluate the performance of ChatGPT (GPT-3.5, GPT-4, o1 models) in optometry and vision science using long-answer questions.
• To explore the feasibility of using LLMs as virtual graders in optometry education.
• To benchmark LLM responses against human graders across various optometry sub-fields.

Main Methods:

• ChatGPT (GPT-3.5, GPT-4, o1) responses to optometry long-answer questions were analyzed.
• LLM performance was compared to human graders, with statistical analyses including mixed-effect models and non-parametric tests.
• Thematic analysis was used to identify response errors across different GPT models and optometry sub-fields.

Main Results:

• All tested GPT models generally produced satisfactory responses (above 50% pass mark), with performance varying significantly by sub-field and model.
• Newer models (GPT-4, o1) significantly outperformed older models (GPT-3.5), and the o1 model produced no thematic errors.
• ChatGPT graders often awarded higher marks than human graders, though this was statistically significant only for GPT-3.5.

Conclusions:

• LLMs demonstrate potential in optometry and vision science education but exhibit inconsistent performance across sub-fields.
• The adoption of LLMs in optometry assessment requires careful consideration of ethical implications, transparency, and the need for judicious adaptation of teaching practices.