Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Temporary Vascular Shunting to Prevent Excessive Ischemia Time for a Fillet-of-Leg Free Flap During External Hemipelvectomy: A Case Report.

Microsurgery·2026
Same author

Generative modeling for RNA splicing prediction and design.

eLife·2026
Same author

Mechanistic understanding of a bifunctional carbonate additive for enhanced performance in lithium-sulfur battery.

Energy storage materials·2026
Same author

Simulated Workflow Feasibility Evaluation of a Web-Based Periorbital Measurement Platform: Development and Usability Study.

JMIR human factors·2026
Same author

Detection of a sequence feature for recursive splicing.

bioRxiv : the preprint server for biology·2026
Same author

Clinical Characteristics of Emergency Visits Related to Recreational Psychedelic Use.

Community mental health journal·2026

Related Experiment Video

Updated: Jun 13, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

519

Prompt engineering to increase GPT3.5's performance on the Plastic Surgery In-Service Exams.

George R Nahass1, Sydney W Chin2, Isabel M Scharf2

  • 1The Craniofacial Center, Division of Plastic, Reconstructive, and Cosmetic Surgery, University of Illinois at Chicago, Chicago, IL 60612, USA; Richard and Loan Hill Department of Biomedical Engineering, University of Illinois at Chicago, Chicago, IL 60612, USA.

Journal of Plastic, Reconstructive & Aesthetic Surgery : JPRAS
|September 10, 2024
PubMed
Summary

ChatGPT (GPT-3.5) showed limited performance on a plastic surgery exam, scoring in the 10th percentile. Prompt engineering and Retrieval Augmented Generation (RAG) did not significantly improve its accuracy, highlighting the need for advanced AI development in medicine.

Keywords:
ChatGPTIn-Service ExaminationPerformance evaluationPlastic surgery residentPrompt engineeringSimulation

More Related Videos

Model Surgical Training: Skills Acquisition in Fetoscopic Laser Photocoagulation of Monochorionic Diamniotic Twin Placenta Using Realistic Simulators
09:51

Model Surgical Training: Skills Acquisition in Fetoscopic Laser Photocoagulation of Monochorionic Diamniotic Twin Placenta Using Realistic Simulators

Published on: March 21, 2018

19.6K
Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions
06:18

Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions

Published on: April 5, 2024

982

Related Experiment Videos

Last Updated: Jun 13, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

519
Model Surgical Training: Skills Acquisition in Fetoscopic Laser Photocoagulation of Monochorionic Diamniotic Twin Placenta Using Realistic Simulators
09:51

Model Surgical Training: Skills Acquisition in Fetoscopic Laser Photocoagulation of Monochorionic Diamniotic Twin Placenta Using Realistic Simulators

Published on: March 21, 2018

19.6K
Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions
06:18

Author Spotlight: Segmentation and VR for Advanced Neurovascular Interventions

Published on: April 5, 2024

982

Area of Science:

  • Medical Education
  • Artificial Intelligence in Medicine
  • Natural Language Processing

Background:

  • Artificial intelligence (AI) tools like ChatGPT (GPT-3.5) are increasingly explored for medical applications.
  • Evaluating AI performance on standardized medical examinations is crucial for understanding their potential and limitations.

Purpose of the Study:

  • To assess the performance of ChatGPT (GPT-3.5) on the 2021 American Society of Plastic Surgeons (ASPS) In-Service Examination.
  • To investigate the impact of prompt engineering (role-playing) and Retrieval Augmented Generation (RAG) on AI accuracy.

Main Methods:

  • ChatGPT (GPT-3.5) was tested using different prompts simulating roles: resident, attending physician, and medical student.
  • Retrieval Augmented Generation (RAG) was implemented using a curated vector database to provide contextual information.
  • Performance was evaluated based on accuracy on the 2021 ASPS Plastic Surgery In-Service Examination.

Main Results:

  • ChatGPT's highest accuracy was 54% when prompted as a "resident."
  • Retrieval Augmented Generation (RAG) did not improve performance, with accuracy remaining at 54.3%.
  • The AI's overall performance placed it in the 10th percentile compared to human test-takers.

Conclusions:

  • Current AI models like ChatGPT (GPT-3.5) demonstrate suboptimal performance on specialized medical examinations.
  • Prompt modifications and RAG alone are insufficient to enhance AI utility for complex medical knowledge tasks.
  • Further research into AI fine-tuning and advanced methodologies is necessary for effective integration into medical practice.