Prompt engineering to increase GPT3.5's performance on the Plastic Surgery In-Service Exams
George R Nahass1, Sydney W Chin2, Isabel M Scharf2
1The Craniofacial Center, Division of Plastic, Reconstructive, and Cosmetic Surgery, University of Illinois at Chicago, Chicago, IL 60612, USA; Richard and Loan Hill Department of Biomedical Engineering, University of Illinois at Chicago, Chicago, IL 60612, USA.
Abstract:
This study assesses ChatGPT's (GPT-3.5) performance on the 2021 ASPS Plastic Surgery In-Service Examination using prompt modifications and Retrieval Augmented Generation (RAG). ChatGPT was instructed to act as a "resident," "attending," or "medical student," and RAG utilized a curated vector database for context. Results showed no significant improvement, with the "resident" prompt yielding the highest accuracy at 54%, and RAG failing to enhance performance, with accuracy remaining at 54.3%. Despite appropriate reasoning when correct, ChatGPT's overall performance fell in the 10th percentile, indicating the need for fine-tuning and more sophisticated approaches to improve AI's utility in complex medical tasks.
