Artificial Intelligence vs Human Clinicians in Esophagogastroduodenoscopy Appropriateness: A Comparative Study Using...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• Large language models (LLMs) are emerging in clinical decision-making.
• Their utility in appropriateness-based indications remains unclear.

Purpose of the Study:

• To compare the appropriateness of esophagogastroduodenoscopy indications by various AI models against human physicians.
• To assess AI performance over time in a real-world survey setting.

Main Methods:

• A vignette-based Italian survey was conducted.
• Five AI models (ChatGPT-4.0, ChatGPT-4.5, Gemini, Claude AI, OpenEvidence) were evaluated at two time points (April and September 2025).
• Performance was compared against gastroenterologists, residents, and general practitioners using established guidelines.

Main Results:

• 135 physicians participated in the survey.
• AI performance demonstrated variability across models and time points.
• Accuracies ranged from 50%-90% in April to 63%-80% in September.
• ChatGPT-4.5 and ChatGPT-4.0 showed superior performance compared to physicians in September.

Conclusions:

• Temporal and prompt variability in AI performance necessitate careful consideration.
• Longitudinal, multirun evaluations are crucial before widespread clinical adoption of AI in appropriateness-based indications.