Assessing Artificial Intelligence as a Diagnostic Support Tool for Surgical Admissions in the Emergency Department

Artificial intelligence (AI) is increasingly being used in healthcare for data analysis and decision support. This study assesses the potential of AI (specifically ChatGPT-4o) as a diagnostic support tool for surgical admissions in the Emergency Department and compared its accuracy with that of on-call surgical trainees.

This was a single-institution retrospective study conducted in December 2023. Primary outcomes comprised the agreement and accuracy of diagnoses between AI and trainees. The secondary outcome measure was the similarity of treatment plans. Agreement was defined as the percentage of cases in which both AI and the surgical trainee concurred on the diagnosis or specific aspects of the management plan. Accuracy was defined in both groups as the proportion of provisional diagnoses that corresponded with the definitive diagnosis, as confirmed by subsequent imaging and/or clinical documentation.

One hundred patients were included in the study. The mean age of presenting cases was 54 (±20), years. Abdominal pain was the most commonly reported symptom in provisional diagnoses (68%). The accuracy of provisional diagnoses compared to CT-confirmed diagnoses was 76% for the AI system and 74% for the surgical trainees (p = 0.744). Substantial agreement between the AI and trainee's initial diagnosis was observed (κ = 0.73). There were no statistically significant differences between AI & surgical trainee in provisional diagnosis (p = 0.754), decisions to bring patients to the operating theatre (p = 0.540) and antibiotic administration (p = 0.122). The overall agreement rates were 85%, 58% & 74% respectively.

Commercially-available AI models demonstrate similar diagnostic ability to junior surgical trainees and may serve as a useful decision-support system. These models could be incorporated into electronic record systems as an adjunct to enhance decision-making.