Evaluating GPT-4 Performance on Plastic Surgery Oral Examination Vignettes: A Quantitative and Qualitative Analysis

Area of Science:

• Medical Artificial Intelligence
• Surgical Education Technology
• Plastic Surgery Clinical Decision Support

Background:

• The application of artificial intelligence (AI) in medicine is expanding, yet its specific utility in plastic and reconstructive surgery requires further investigation.
• Plastic surgery's complex decision-making processes present unique challenges for AI integration.
• This study explores GPT-4's capabilities in responding to simulated plastic surgery board examinations.

Purpose of the Study:

• To evaluate GPT-4's performance on American Board of Plastic Surgery (ABPS) oral board-style clinical vignettes.
• To assess GPT-4's potential as a tool for decision-making and education in plastic surgery.
• To identify areas where AI can supplement current surgical training and practice.

Main Methods:

• Twelve diverse clinical vignettes from plastic surgery subspecialties were presented to GPT-4.
• Responses were evaluated by two board-certified plastic surgeons using a 0-4 scoring rubric across six domains.
• Interrater reliability was assessed, and qualitative interviews were conducted.

Main Results:

• GPT-4 achieved average scores between 2.3 and 3.7, with overall passing rates of 91.7% and 83.3% for the two raters.
• Diagnosis and complication management were the highest-scoring domains, while case introductions scored lowest.
• Qualitative analysis highlighted GPT-4's accuracy in guideline-based scenarios but noted limitations in trauma triage.

Conclusions:

• GPT-4 demonstrates significant alignment with board-level clinical reasoning in structured plastic surgery scenarios.
• Limitations in heuristic judgment and adaptability necessitate further AI optimization using specialized datasets and rubrics.
• GPT-4 holds potential as a valuable supplementary tool for surgical education and oral board preparation.