A Comparative Evaluation of Large Language Models on Pediatric Board-Style Examinations

To evaluate the performance of 4 leading large language models (LLMs) on pediatric board-style examinations and to assess whether question characteristics are associated with response accuracy.

This study evaluated LLM platforms using the 2023 and 2024 editions of the American Academy of Pediatrics PREP® Self-Assessment. Under standardized testing conditions, all examination items were entered sequentially into each model, and responses were scored for accuracy. Mixed-effects logistic regression was applied to examine associations between question characteristics and correctness, accounting for clustering by question ID.

Across both examination years, all models achieved high overall accuracy exceeding typical human passing thresholds. ChatGPT-5 demonstrated numerically the highest accuracy (89.1-89.6%), followed by Perplexity (86.1-86.6%), Claude Opus 4.1 (81.8-84.0%), and Gemini 2.5 (78.4-86.3%). Pairwise posthoc comparisons revealed no statistically significant differences among models after adjustment for multiple testing (all adjusted P > .90). Regression analysis identified the presence of tables as a significant predictor of reduced accuracy (odds ratio 0.52, 95% CI 0.29-0.91, P = .022), whereas question length and the presence of images were not significantly associated with performance. No significant interactions between model identity and question characteristics were observed.

Current market-leading LLMs exhibit strong and broadly comparable performance on pediatric board-style examinations. However, persistent difficulty with structured and tabular data represents a shared limitation that may affect reliability in pediatric educational and clinical contexts. While LLMs are valuable tools for pediatric examination preparation and self-assessment, continued refinement is needed to improve their interpretation of structured medical information.