Evaluating Large-Language Models Against Providers on Surgical Diagnostic Reasoning Tasks

Large language models (LLMs) have shown efficacy in surgery-related tasks such as literature screening and performing well on-board examination; however, their utility in diagnostic reasoning remains unclear. We assessed the accuracy and internal consensus of residents against OpenAI's GPT-3.5 and GPT-4 in constructing differential diagnoses.

This single-institution study involved a military general surgery program. Eleven general surgery residents develop a five-ranked differential diagnosis across 10 case vignettes. GPT-3.5 and GPT-4 were each queried 30 times per vignette. Four attending surgeons created their own five-ranked differential diagnoses. We used a Borda count to combine those lists into a gold-standard reference list for accuracy comparison.

Junior (odds ratio [OR]: 1.62, 95% confidence interval [CI]: 1.17-2.25) and senior residents (OR: 1.76, 95% CI: 1.23-2.52) outperformed GPT-4 (OR: 0.94, 95% CI: 0.78-1.14) and GPT 3.5 in accuracy. After accounting for interaction effects from vignette, performance was similar across groups, with residents either outperforming or underperforming the LLMs dependent on vignette. Residents outperformed LLMs on gastrointestinal cases with high ambiguity, whereas LLMs outperformed residents on nongastrointestinal cases regardless of their ambiguity. LLMs (GPT 3.5: OR: 1.93, 95% CI: 1.17-3.31; GPT 4: OR: 2.24, 95% CI: 1.28-3.94) displayed more internal consensus among diagnoses compared to residents (junior residents: OR: 1.07, 95% CI: 0.69-1.66; senior residents: OR: 1.44; 95% CI: 0.90-2.31), and attendings.

We conclude that the ideal use of LLMs in surgical education is context specific and may be particularly useful when residents are less experienced with the pathology.