Related Experiment Video
Updated: Apr 18, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
1.3K
Evaluating Large-Language Models Against Providers on Surgical Diagnostic Reasoning Tasks
Taj Keshav1, David Chow2, Tiffany Kippenberger2
1Department of Preventive Medicine and Biostatistics, Uniformed Services University of the Health Sciences, Bethesda, Maryland.
The Journal of Surgical Research
|April 16, 2026
Summary
General surgery residents demonstrated superior accuracy in differential diagnoses compared to large language models (LLMs). LLMs showed higher internal consensus, suggesting context-specific utility in surgical education.
Area of Science:
- Medical Education
- Artificial Intelligence in Medicine
- Surgical Training
Background:
- Large language models (LLMs) show promise in surgery-related tasks but their diagnostic reasoning utility is unclear.
- Assessing LLM accuracy against surgical residents in differential diagnosis construction is crucial.
Purpose of the Study:
- To evaluate the accuracy and internal consensus of surgical residents versus GPT-3.5 and GPT-4 in generating differential diagnoses.
- To compare LLM diagnostic performance with that of junior residents, senior residents, and attending surgeons.
Main Methods:
- Eleven general surgery residents developed differential diagnoses for 10 case vignettes.
- GPT-3.5 and GPT-4 were queried 30 times per vignette.
- Attending surgeons' diagnoses were combined with resident diagnoses using a Borda count to create a gold standard.
Main Results:
- Residents outperformed GPT-4 and GPT-3.5 in diagnostic accuracy.
- LLMs exhibited greater internal consensus than residents.
- Performance varied by case ambiguity and specialty; residents excelled in ambiguous gastrointestinal cases, while LLMs led in non-gastrointestinal cases.
Conclusions:
- LLM utility in surgical education is context-dependent.
- LLMs may be most beneficial for less experienced residents facing unfamiliar pathologies.
- Integrating LLMs requires careful consideration of their strengths and weaknesses in diagnostic reasoning.
