Related Experiment Video
Updated: Jan 10, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Evaluating the performance of large language models on the ASPS In-Service Examination: A comparative analysis with
Ramin Shekouhi1, Mary M Holohan1, Oygul Mirzalieva2
1Division of Plastic and Reconstructive Surgery, Department of Surgery, Louisiana State University Health Sciences Center, New Orleans, LA, USA.
Abstract:
The emergence of large language models (LLMs) has raised critical questions about their potential roles in surgical education. This study aims to evaluate the accuracy and comparative performance of three leading LLMs including ChatGPT 4.0, DeepSeek V3, and Gemini 2.5, on the American Board of Plastic Surgery Plastic Surgery In-Service Training Examination (PSITE) across a 20-year period. Our results showed that ChatGPT achieved the highest overall accuracy (75.0%), followed closely by DeepSeek (74.8%) and Gemini (74.5%), with no significant differences between models (p>0.05). When benchmarked against normative data, DeepSeek reached the highest percentile ranks (81st among residents, 89th among practitioners), followed by ChatGPT (78th and 84th), and Gemini (72nd and 90th), without significant differences in rankings across LLMs (p > 0.05). In conclusion, Modern LLMs demonstrate consistent and high-level performance on the PSITE, frequently exceeding the median performance of plastic surgery residents and practitioners.
