When Wrong Answers Matter: Consequence-Weighted Evaluation of Large Language Models for ERCP Triage

Area of Science:

• Artificial Intelligence in Medicine
• Gastroenterology
• Clinical Decision Support

Background:

• Large language models (LLMs) are increasingly used for clinical recommendations.
• Their accuracy in translating biliary guidelines for procedural triage is uncertain.
• Evaluating LLM performance in ERCP indication for choledocholithiasis is crucial.

Purpose of the Study:

• To assess the diagnostic accuracy of next-generation LLMs for ERCP indication in suspected choledocholithiasis.
• To evaluate the impact of LLM errors on clinical workflow and patient safety.
• To compare the performance of GPT-5.5, Gemini 3.0 Pro, and Claude 4 Opus.

Main Methods:

• A cross-sectional in-silico diagnostic accuracy study.
• 100 synthetic vignettes mapped to ASGE/ESGE guidelines (45 indicated, 55 non-indicated).
• LLMs queried using zero-shot prompting; outcomes included accuracy, sensitivity, specificity, kappa, error phenotype, and simulated under-triage delay.

Main Results:

• GPT-5.5 achieved highest accuracy (96.0%), followed by Gemini 3.0 Pro (90.0%) and Claude 4 Opus (84.0%).
• GPT-5.5 showed near-perfect agreement (kappa=0.92); Claude 4 Opus had weaker agreement (kappa=0.68) and most under-triage errors (n=9).
• Claude 4 Opus resulted in the largest simulated delay burden (163.8 hours/100 vignettes).

Conclusions:

• Next-generation LLMs can approximate guideline-based ERCP triage for choledocholithiasis.
• Clinically significant differences exist, particularly concerning under-triage errors and procedural delays.
• GPT-5.5 demonstrated a balanced profile, but human supervision is essential to mitigate under-triage risks.