Evaluating the Performance of Large Language Models in Vascular Surgery Cases: Cross-Sectional Study
Related Experiment Video
Updated: Nov 8, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Asanka Wijetunga1,2, Yunyi Wang1, Chantell Yaghi1
1Westmead Hospital, Cnr Darcy and Hawkesbury Road, Westmead, New South Wales, 2145, Australia, 61 8890 5555 ext 9.
Background:
Large language models (LLMs) such as ChatGPT, Gemini, and Claude are increasingly used by clinicians, yet their accuracy, safety, and consistency in clinical cases remain poorly defined. Most studies assess LLMs using multiple-choice questions rather than free-response reasoning.
Objective:
The primary objective of this study was to evaluate the performance and safety of 3 widely used LLMs in realistic vascular surgery scenarios in key domains such as diagnosis, investigation, and multidisciplinary and multimodal management. The secondary objective of this study was to evaluate the safety of the models' outputs.
Methods:
A total of 42 fictitious vascular cases across 5 major pathologies (acute limb ischemia [ALI], aortopathy, chronic limb ischemia, diabetic foot infection, and extracranial cerebrovascular disease) were independently entered into GPT-5, Gemini 2.5, and Claude Sonnet 4.5 using standardized prompts. Each model answered structured questions covering diagnosis, investigation, and management (both operative and nonoperative). Responses were scored out of 20 by a panel using a predefined rubric. Each plan's overall safety was separately assessed. Comparative analyses used 2-tailed t tests, ANOVA, and multivariable logistic regression.
Results:
Mean composite scores by model were 86.5% for ChatGPT, 83.5% for Gemini, and 88.0% for Claude, whereas scores by disease were 82.7% for ALI, 83.8% for aortopathy, 85.9% for chronic limb ischemia, 86.1% for diabetic foot infection, and 80.7% for extracranial cerebrovascular disease (P>.05 in each case). Unsafe plans were generated in 11.9%, 23.8%, and 7.1% of cases by ChatGPT, Gemini, and Claude, respectively. On multivariable analysis, independent predictors of unsafe outputs were lower composite score (odds ratio [OR] 0.471, 95% CI 0.306-0.722; P=.001), higher word count (OR 1.003, 95% CI 1.001-1.005; P=.001), and ALI (OR 20.9, 95% CI 3.79-61.8; P<.001).
Conclusions:
Our findings demonstrate LLMs' promise in managing routine vascular surgery cases. However, their inconsistent safety profiles and ethical limitations preclude unsupervised clinical use. Rigorous specialty-specific validation is essential before they may be integrated into routine practice.