Related Experiment Video
Updated: Jun 12, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Artificial intelligence large language model scores highly on focused practice designation in metabolic and bariatric
A Sanders1,2, R Lim3, D Jones4
1Department of Surgery, Beth Israel Deaconess Medical Center, Boston, MA, USA.
Background:
Artificial intelligence models such as ChatGPT (Open AI) have performed well on the exams of various medical and surgical fields. It is not yet known how ChatGPT performs on similar metabolic and bariatric surgery (MBS) questions.
Objective:
Assess the performance of ChatGPT on Focused Practice Designation in Metabolic and Bariatric Surgery board-style questions.
Setting:
United States.
Methods:
Questions obtained from the largest commercially available bank of FPD-MBS practice questions were entered into ChatGPT-4, as is, without prior training. We assessed the overall percentage correct as well as the percentage correct within each of the five American Board of Surgery (ABS) question categories. One-way ANOVA was used to determine if the frequency of correct answers differed between categories.
Results:
Out of 255 questions, ChatGPT-4 correctly answered 189 (74.1%). Between the five question categories there was no difference between the frequency of correct answers (p = 0.22). It did not matter if questions were entered individually or in groups of up to 10.
Conclusion:
Without prior training, ChatGPT-4 scored highly when evaluated on the largest practice question bank for the FPD-MBS exam.
