Large Language Models Triage of Retina Patient Emergency Telephone Calls: A Pilot Study

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Diagnostics

Background:

• On-call telephone triage is crucial for managing urgent ophthalmic conditions.
• Large language models (LLMs) are increasingly explored for medical applications.
• Evaluating AI diagnostic accuracy in real-time clinical scenarios is essential.

Purpose of the Study:

• To compare the diagnostic and management accuracy of LLM chatbots against human experts in outpatient retina triage during telephone emergencies.
• To assess the performance variability among different LLM chatbots.

Main Methods:

• Ten simulated retina emergency cases were presented to four LLM chatbots, three vitreoretinal surgery fellows, and three certified ophthalmic technicians.
• Diagnoses and triage recommendations were collected and graded for accuracy by human evaluators.
• Statistical analysis was performed to compare the performance of chatbots and humans.

Main Results:

• Human experts achieved significantly higher accuracy in diagnosis (95%) and follow-up recommendations (85%) compared to overall chatbots (76.7% and 70%, respectively).
• ChatGPT and Claude demonstrated non-inferior diagnostic (90% and 83.3%) and recommendation (93.3% and 90%) accuracy compared to humans.
• Gemini and Meta chatbots performed significantly worse than humans in both diagnostic and management tasks.

Conclusions:

• While human experts currently outperform LLM chatbots in retina emergency triage, specific models like ChatGPT and Claude show comparable accuracy.
• The variable performance of LLMs highlights the need for further validation before widespread clinical implementation.
• Certain LLMs may serve as valuable assistive tools for managing urgent medical telephone calls in the future.