Related Experiment Videos
Reliability, Quality, and Readability of Freely Available Large Language Models for Clinical Decision Support in
Murat Coskun1, Fatih Berke Inan2, Bahaeddin Umur Aka1
1Department of General Surgery, Sakarya Training and Research Hospital, Sakarya 54100, Turkey.
Abstract:
Background/Objectives: Perforated peptic ulcer (PPU) is a common surgical emergency in which each hour before surgery increases mortality, and the decision is often made by a single surgeon on call. Large language models (LLMs) may offer rapid decision support, but their reliability in this setting is unknown. We compared five freely accessible LLMs in a blinded, multidimensional design. Methods: Twenty-four structured questions derived from the WSES 2020 guideline, covering the preoperative, intraoperative, and postoperative phases, were posed to the free versions of ChatGPT (GPT-5.3 Instant), Claude (Sonnet 4.6), DeepSeek-V3, Gemini (2.0 Flash), and Grok 3 during April and May 2026, yielding 120 responses. Two general surgeons blinded to model identity scored scientific reliability with the modified DISCERN instrument (mDISCERN, 8-40) as the primary measure and overall quality with the Global Quality Score (GQS, 1-5) as a secondary measure. Because every question was answered by all five models, the Friedman test was primary. Results: Models differed significantly in reliability (Friedman chi-square(4) = 71.35, p < 0.001, Kendall W = 0.743) and separated into two tiers: Claude (mDISCERN 37.83), Grok 3 (36.08), and DeepSeek-V3 (34.92) scored higher than ChatGPT (30.62) and Gemini (29.48). All between-tier comparisons were significant, whereas no within-tier comparison reached significance. Inter-rater agreement was moderate (mDISCERN ICC(2,k) = 0.647) and internal consistency high (Cronbach alpha 0.867 and 0.916). Reliability correlated positively with response length (rho = 0.717) and negatively with readability (rho = -0.452). Conclusions: Freely accessible LLMs vary substantially in reliability for PPU, so model choice has clinically meaningful consequences. This variation is specific to the models, versions, and access tier evaluated, and no model should be used for emergency surgical decision support without independent, setting-specific validation.
Related Concept Videos
Peptic Ulcer Disease III: Clinical Manifestations and Diagnostic Studies
Few clinical manifestations differentiate gastric ulcers from duodenal ulcers. Distinctions in the location, timing, and pain relief are crucial for healthcare providers in differentiating between gastric and duodenal ulcers during clinical assessments.
Peptic Ulcer Disease III: Clinical Manifestations and Complications
Peptic Ulcer
Peptic Ulcer Disease I: Introduction
An acute ulcer, marked by superficial erosion and minimal inflammation, swiftly resolves upon identifying and addressing the underlying cause. In contrast, a chronic ulcer persists, potentially eroding through the muscular wall and forming fibrous tissue.
Peptic ulcers can also be...
Peptic Ulcer Disease I: Introduction
Peptic Ulcer Disease II: Pathophysiology