Related Experiment Video
Updated: May 26, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
A Comparative Analysis of AI-Language Models' MCQ Performance versus Medical Students Across Different Pediatric
Olena Bolgova1, Volodymyr Mavrych1, Eyad Almidani2
1Department of Anatomy, College of Medicine, Alfaisal University, Riyadh, Saudi Arabia.
Background:
Large Language Models (LLMs) are increasingly used in medical education, yet their performance in specialized fields like pediatrics remains understudied.
Objective:
To evaluate and compare the performance of four leading LLMs on a standard set of pediatric multiple-choice questions (MCQs) and benchmark their results against those of medical students.
Methods:
We assessed 4 leading LLMs: Copilot (Microsoft), Claude (Anthropic), ChatGPT (OpenAI), and Gemini (Google), on 120 MCQs from six pediatric topics: Pulmonology, Developmental Diseases, Infectious Diseases, General Pediatrics, Neonatology, and Nephrology. Each LLM attempted the questions three times to evaluate consistency. Medical student performance on the same questions served as a benchmark, with random responses providing baseline control.
Results:
The LLMs demonstrated an average accuracy of 80.4±10.9%, outperforming medical students (72.1±13.1%) by 8.3% and exceeding random responses (20.5±6.2%) by nearly fourfold. Copilot (84.5±8.3%) and Claude (84.2±9.9%) achieved the highest accuracy, followed by GPT-4o (80.9±9.1%) and Gemini (72.0±18.9%). Among all LLM-to-student comparisons, only GPT-4o showed a statistically significant difference in accuracy (χ2 = 4.277, p = 0.039), outperforming students by 8.8%. LLMs excelled in Pulmonology (96.3%), Developmental Diseases (86.3%), and Infectious Diseases (85%) while struggling with Nephrology (67.5%) and Neonatology (69.7%). Only 50% (60/120) of questions were correctly answered by all LLMs across all attempts, while 6.7% (8/120) were never answered correctly.
Conclusion:
Modern LLMs demonstrate proficiency in pediatric knowledge that generally exceeds medical student performance, though with varying consistency across topics. These findings suggest LLMs may serve as valuable supplementary tools in medical education while highlighting the need for further improvements in specialized medical domains like nephrology and neonatology.