Related Experiment Video
Updated: Jun 29, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Comparative Evaluation of Large Language Models for Surgical Case Creation
Connie Y Gan1, Megan Chan2, Nitasha Sharma3
1Department of Surgery, Stanford University, Stanford, California; Department of Surgery, Oregon Health and Science University, Portland, Oregon.
Objective:
To evaluate and compare the performance of five large language models (LLMs)-ChatGPT-4o (OpenAI), Claude-3.5-sonnet (Anthropic), Gemini-2.0-flash-001 (Google), Llama-3.2 (Meta), and DeepSeek-r1 (DeepSeek) in generating trauma surgery case scenarios for ENTRUST, a virtual simulation platform designed to teach and assess clinical decision-making in surgical trainees.
Design:
Comparative, experimental analysis using a standardized prompt applied to each LLM alongside surgical reference materials: American Board of Surgery General Surgery trauma entrustable professional activity (EPA) definitions, trauma surgery references, ENTRUST platform description and case template. Outputs were blinded and independently evaluated by content experts. Expert perceptions, quality metrics adapted from HumanELY (relevance, coverage, coherence, and lack of harm), usability, and Flesch-Kincaid Grade Level readability were evaluated.
Setting:
Multi-institutional study.
Participants:
Four board-certified trauma surgeons from multiple U.S. institutions served as content experts.
Results:
Claude-3.5-sonnet outperformed other LLMs across most metrics, with a significant mean total modified HumanELY score of 52.5/70 (p = 0.04). Gemini-2.0-flash-001 (46.7/70), ChatGPT-4o (45.8/70), and DeepSeek-r1 (45.8/70) performed similarly, while Llama-3.2 scored lowest (39.8/70). Readability analysis showed Flesch-Kincaid Grade Levels ranging from 9.9 to 12.1. Claude required the least editing (22.5%) to make outputs useful, while Llama required the most (66.3%).
Conclusions:
LLM performance in surgical case generation varies substantially across models. Claude-3.5-sonnet demonstrated superior performance in generating high-quality trauma surgery cases, suggesting that careful selection and optimization of LLMs is crucial for their effective implementation into surgical education. In addition, this study introduces a modified HumanELY framework as a structured, human-based tool that educators can use to evaluate the quality of AI-generated surgical education content.