Accuracy of a Large Language Model as a new tool for optometry education

Area of Science:

• Optometry Education
• Artificial Intelligence in Healthcare
• Ophthalmology Research

Background:

• Large Language Models (LLMs) like ChatGPT are increasingly adopted by students and researchers.
• The confident tone of LLMs can obscure limitations in scientific and clinical query responses.
• Unsupervised AI integration in optometry education may impede clinical knowledge and skill acquisition.

Purpose of the Study:

• To evaluate the accuracy and reference quality of ChatGPT responses in optometry.
• To assess student and expert perceptions of ChatGPT's utility in academic optometry.
• To investigate the impact of query specificity on AI response quality.

Main Methods:

• ChatGPT was queried on contact lenses, low vision, and binocular vision topics.
• Responses were evaluated by experts and students for accuracy (0-10 scale).
• References provided by ChatGPT were assessed for precision and relevance.

Main Results:

• Median accuracy scores ranged from 6-8 (experts) and 7.5-9 (students).
• More specific queries received lower accuracy scores from both groups (p<0.001).
• Only 24% of references were accurate and 19.3% were relevant.

Conclusions:

• Expert appraisal of ChatGPT responses and references is crucial for academic and research use.
• Proactive measures are needed to address LLM limitations as their use expands.
• Responsible integration of AI tools is essential for optometry education.