Multiple large language models versus experienced physicians in diagnosing challenging cases with gastrointestinal...

Area of Science:

• Medical Diagnostics
• Artificial Intelligence in Healthcare
• Gastroenterology

Background:

• Physicians increasingly consult large language models (LLMs) for complex diagnostic challenges.
• The diagnostic capabilities of LLMs compared to human experts remain an area of active research.

Purpose of the Study:

• To compare the diagnostic accuracy of LLMs against human gastroenterologists for challenging gastrointestinal cases.
• To evaluate the effectiveness of LLMs as a diagnostic aid in clinical practice.

Main Methods:

• An offline dataset of 67 challenging gastrointestinal cases was utilized.
• Seven LLMs and 22 gastroenterologists were tasked with providing diagnoses for the cases.
• Diagnostic coverage and instructive diagnoses were compared between LLMs and human physicians.

Main Results:

• Claude 3.5 Sonnet provided the highest proportion of instructive diagnoses (76.1%), significantly outperforming all participating gastroenterologists.
• Claude 3.5 Sonnet's diagnostic coverage rate (76.1%) was significantly higher than that of gastroenterologists using traditional resources (45.5%).
• The performance difference was statistically significant (p < 0.05 for gastroenterologist comparison, p < 0.001 for resource comparison).

Conclusions:

• Advanced LLMs, such as Claude 3.5 Sonnet, demonstrate superior diagnostic capabilities for challenging gastrointestinal cases compared to human experts.
• LLMs can serve as valuable, time-saving, and cost-effective tools to assist gastroenterologists in complex diagnostic scenarios.
• The findings suggest a potential role for AI in enhancing diagnostic accuracy and efficiency in specialized medical fields.