Related Experiment Video
Updated: May 30, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Performance of Chat Generative Pre-Trained Transformer on Personal Review of Learning in Obstetrics and Gynecology
Adam Cohen1, Jersey Burns2, Martina Gabra1
1From the Departments of Urology.
Objectives:
Chat Generative Pre-Trained Transformer (ChatGPT) is a popular natural-language processor that is able to analyze and respond to a variety of prompts, providing eloquent answers based on a collection of Internet data. ChatGPT has been considered an avenue for the education of resident physicians in the form of board preparation in the contemporary literature, where it has been applied against board study material across multiple medical specialties. The purpose of our study was to evaluate the performance of ChatGPT on the Personal Review of Learning in Obstetrics and Gynecology (PROLOG) assessments and gauge its specialty specific knowledge for educational applications.
Methods:
PROLOG assessments were administered to ChatGPT version 3.5, and the percentage of correct responses was recorded. Questions were categorized by question stem order and used to measure ChatGPT performance. Performance was compared using descriptive statistics.
Results:
There were 848 questions without visual components; ChatGPT answered 57.8% correct (N = 490). ChatGPT performed worse on higher-order questions compared with first-order questions, 56.8% vs 60.5%, respectively. There were 65 questions containing visual data, and ChatGPT answered 16.9% correctly.
Conclusions:
The passing score for the PROLOG assessments is 80%; therefore ChatGPT 3.5 did not perform satisfactorily. Given this, it is unlikely that the tested version of ChatGPT has sufficient specialty-specific knowledge or logical capability to serve as a reliable tool for trainee education.
