Solving Complex Pediatric Surgical Case Studies: A Comparative Analysis of Copilot, ChatGPT-4, and Experienced Pediatric Surgeons' Performance

Richard Gnatzy ¹, Martin Lacher ¹, Michael Berger ²

¹Department of Pediatric Surgery, Leipzig University, Leipzig, Germany.
²Department of Pediatric Surgery, University Hospital Essen, Essen, Germany.
³Department of Pediatric Surgery, University Medical Centre Mannheim, Mannheim, Germany.
⁴Department of Pediatric Surgery, Hospital Bremen-Mitte, Bremen, Germany.
⁵Department of Pediatric Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
⁶Centre for Pediatric Surgery, Department of Pediatric Surgery and Urology, University Hospital Giessen-Marburg, Baldingerstraße, Marburg, Germany.
⁷Centre for Pediatric Surgery, Department of Pediatric Surgery, University Hospital Giessen-Marburg, Giessen, Germany.
⁸Department of Pediatric Surgery, University of Helsinki Children's Hospital Unit of Pediatric Surgery, Helsinki, Finland.
⁹Department of Pediatric Surgery, Karolinska University Hospital, Stockholm, Sweden.
¹⁰Department of Women's and Children's Health, Karolinska Institutet, Stockholm, Sweden.
¹¹Department of Surgery, Division of Pediatric Surgery, Washington University School of Medicine, St. Louis, Missouri, United States.

⁰Department of Pediatric Surgery, Leipzig University, Leipzig, Germany.

European Journal of Pediatric Surgery : Official Journal of Austrian Association of Pediatric Surgery ... [Et Al] = Zeitschrift Fur Kinderchirurgie +

|

March 5, 2025

View abstract on PubMed

Summary

Large language models (LLMs) like ChatGPT-4 and Copilot show limited accuracy in pediatric surgery diagnosis and clinical questions. While AI has potential, human surgeons still outperform these advanced models.

Area Of Science

Medical Artificial Intelligence
Pediatric Surgical Diagnostics
Clinical Decision Support Systems

Background

Large language models (LLMs) are advancing rapidly across sectors, including medicine.
The application and efficacy of LLMs in pediatric surgery are not well-established.
Assessing AI capabilities in complex pediatric surgical cases is crucial for future integration.

Purpose Of The Study

To evaluate the diagnostic and clinical question-answering abilities of ChatGPT-4 and Microsoft Copilot in pediatric surgery.
To compare the performance of these artificial intelligence (AI) models against experienced pediatric surgeons.
To assess the completeness and accuracy of AI-generated diagnostic recommendations.

Main Methods

Utilized 13 complex clinical case vignettes of classic pediatric surgical diseases.
Compared AI model responses (ChatGPT-4, Copilot) with those of a cohort of pediatric surgeons.
Pediatric surgeons rated AI diagnostic recommendations; statistical analyses determined performance differences.

Main Results

Pediatric surgeons achieved the highest performance score (68.8%), followed by ChatGPT-4 (52.1%) and Copilot (47.9%).
Statistically significant performance differences were observed between AI models and human surgeons (p < 0.01).
ChatGPT-4 showed superior differential diagnosis generation compared to Copilot (p < 0.05); AI recommendations were rated as average.

Conclusions

Current AI models demonstrate significant limitations in accuracy and reliability for pediatric surgical clinical decision-making.
While LLMs show promise, their current capabilities do not match experienced human surgeons in this specialized field.
Further research and development are necessary to enhance AI performance and validate its clinical utility in pediatric surgery.

Related Concept Videos

Comparing the Survival Analysis of Two or More Groups

Survival analysis is a cornerstone of medical research, used to evaluate the time until an event of interest occurs, such as death, disease recurrence, or recovery. Unlike standard statistical methods, survival analysis is particularly adept at handling censored data—instances where the event has not occurred for some participants by the end of the study or remains unobserved. To address these unique challenges, specialized techniques like the Kaplan-Meier estimator, log-rank test, and...