Diagnostic Ambiguity in Appendicitis: Comparing Failure Modes of Large Language Models and Surgeons

Area of Science:

• Artificial Intelligence in Medicine
• Diagnostic Accuracy
• Emergency Medicine

Background:

• Triage of suspected acute appendicitis is challenging in atypical presentations, such as pediatric, geriatric, or cases with subtle inflammatory markers.
• Frontier large language models (LLMs) are being explored for emergency decision-making, but their clinical readiness requires rigorous evaluation beyond simple vignette comparisons.
• This study addresses the need to assess LLM diagnostic behavior, triage safety, and error patterns in standardized, diagnostically complex appendicitis scenarios.

Purpose of the Study:

• To evaluate and compare the diagnostic accuracy and triage safety of frontier LLMs against general surgery specialists in simulated appendicitis cases.
• To characterize the error profiles and failure modes of both AI systems and human experts in ambiguous diagnostic scenarios.

Main Methods:

• A comparative, cross-sectional diagnostic-accuracy simulation study was conducted using 150 standardized emergency department vignettes (90 appendicitis, 60 mimics), focusing on difficult cases.
• Six frontier AI systems and two board-certified general surgery specialists independently evaluated each vignette using a standardized triage prompt.
• Key outcomes measured included diagnostic accuracy, sensitivity, specificity, inter-rater agreement, triage-priority accuracy, and a qualitative error taxonomy.

Main Results:

• Significant differences in diagnostic performance were observed across AI systems and specialists (P < 0.001).
• GPT-5 demonstrated the highest diagnostic accuracy (95.3%), outperforming GPT-4o (82.0%) and slightly exceeding specialist consensus (92.7%).
• GPT-5 maintained high accuracy (93.8%) in pediatric and geriatric cases, surpassing specialists (90.0%) and GPT-4o (71.2%). Distinct AI failure modes, like hallucinations, were noted.

Conclusions:

• Performance of frontier LLMs in simulated appendicitis diagnosis varies significantly, with GPT-5 showing notable accuracy.
• The study highlights distinct human and AI failure modes in diagnostically ambiguous situations, emphasizing the need for careful interpretation of AI performance.
• Findings support supervised, diagnosis-specific AI evaluation rather than autonomous deployment in emergency triage settings.