Related Experiment Video
Updated: Feb 17, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Can artificial intelligence pass the test? Evaluating chatbot scores on pediatric gastroenterology board-style
Annette G Roberts1,2, Reshma Patel1, Sharmilaa Babu1
1Division of Gastroenterology, Hepatology, and Nutrition Duke University Hospital Durham North Carolina USA.
Objectives:
The American Academy of Pediatrics (AAP) Pediatrics Review and Education Program (PREP)® Gastroenterology (GI) Self-Assessments help pediatric gastroenterologists and trainees prepare for subspecialty board exams by providing peer-reviewed questions and critiques based on American Board of Pediatrics content specifications. These assessments test knowledge of material aligned with the pediatric gastroenterology board exams. While artificial intelligence (AI) chatbots have passed various medical board exams, their ability to pass the pediatric GI boards remains untested. This study assesses the performance of Microsoft Copilot and OpenAI ChatGPT-3.5 and 4o on the 2022-2024 AAP PREP® GI Self-Assessments.
Methods:
A total of 216 AAP PREP® GI Self-Assessment questions from 2022 to 2024 were entered into three AI chatbots (Microsoft Copilot, OpenAI ChatGPT-3.5, and ChatGPT-4o). Scores were compared with the passing score (> 65%) and first-time test takers' scores from the AAP for 2022-2024.
Results:
OpenAI ChatGPT-4o and Microsoft Copilot scored above 65% (pass) on all three PREP® GI Self-Assessments from 2022 to 2024. OpenAI ChatGPT-3.5 passed the 2023 and 2024 assessments but did not pass the 2022 assessment. The chatbots collectively scored best in anatomy, motility, and mouth and esophageal disorders, and scored poorly in physiology, pharmacology, liver, stomach and duodenum disorders.
Conclusions:
OpenAI ChatGPT-4o and Microsoft Copilot consistently passed the PREP® GI Self-Assessments from 2022 to 2024, showing potential for good performance on the pediatric GI boards. OpenAI ChatGPT-3.5 had limitations, passing only the 2023 and 2024 assessments. Overall, advanced AI chatbots show potential to pass the Pediatric GI board exam.
Related Concept Videos
01:29Assessment of the Gastrointestinal System II: Health Perception Pattern
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
01:17Assessment of the Gastrointestinal System I: Subjective Data
Health History
The initial step in assessing the GI system is obtaining a comprehensive health history. This includes inquiring about the patient's history or presence of problems...