Evaluating multiple large language models on orbital diseases.

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Education

Background:

• Human learning involves time-consuming error correction and experience.
• Large language models (LLMs) like ChatGPT offer human-like responses to diverse problems.
• LLMs can potentially assist in complex medical learning processes.

Purpose of the Study:

• To evaluate the effectiveness of LLMs as educational assistants for orbital disease inquiries.
• To compare the performance of various LLMs in answering orbital disease-related questions.
• To assess the potential of LLMs to support ophthalmology trainees and practitioners.

Main Methods:

• A dataset of 100 orbital disease questions and answers from resident/student exams was compiled.
• Five LLMs (GPT-4, GPT-3.5, PaLM2, Claude 2, SenseNova) were tested.
• The top-performing LLM was compared against ophthalmologists and medical students.

Main Results:

• GPT-4 and PaLM2 showed superior average correlation among tested LLMs.
• GPT-4 demonstrated the broadest accurate response spectrum and highest average score.
• GPT-4's performance exceeded medical students but was below ophthalmologists in accuracy.

Conclusions:

• GPT-4 shows significant potential as an assistant in the orbital domain of ophthalmology.
• LLMs, particularly GPT-4, can serve as valuable supplementary tools for medical education and practice.
• Further LLM training may enhance their utility for ophthalmologists and medical students.