Assessment of Differential Diagnoses for Oculoplastics Cases Produced by Large Language Models

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Diagnostics

Background:

• Accurate differential diagnoses are crucial for effective oculoplastic patient care.
• Evaluating the diagnostic capabilities of emerging AI technologies like large language models (LLMs) is essential.

Purpose of the Study:

• To assess the accuracy of six distinct large language models in generating differential diagnoses for oculoplastic conditions.
• To compare the performance of various LLMs against expert-curated diagnoses.

Main Methods:

• Twenty oculoplastic cases from EyeRounds.org were used to generate differential diagnoses with six LLMs: ChatGPT 3.5, ChatGPT 4.0, OcuSmart/EyeGPT, Google Gemini 1.5, Claude 3.5, and Microsoft CoPilot.
• LLM outputs were evaluated based on top diagnosis match rate, inclusion of correct diagnosis, recall, and precision compared to expert differentials.

Main Results:

• OcuSmart/EyeGPT achieved the highest top diagnosis match rate (85%).
• Claude 3.5 showed the highest correct diagnosis inclusion and recall (100% and 55%, respectively).
• Google Gemini provided the most precise differentials (43%), though Claude 3.5 generated less concise lists. Performance varied by case type.

Conclusions:

• LLMs demonstrate significant potential for aiding in oculoplastic case diagnosis.
• OcuSmart/EyeGPT and Claude 3.5 are strong performers for diagnosis and recall, while ChatGPT 3.5, OcuSmart/EyeGPT, and Gemini offer concise differentials.
• Further validation and integration studies are needed to incorporate LLMs into clinical practice.