Performance of DeepSeek and GPT Models on Pediatric Board Preparation Questions: Comparative Evaluation

Area of Science:

• Artificial intelligence in medical education
• Large language models (LLMs) for healthcare

Background:

• Limited research evaluates AI performance on standardized pediatric assessments.
• This study addresses the need to assess AI capabilities in pediatric board preparation.

Purpose of the Study:

• To evaluate and compare the performance of three leading LLMs on pediatric board examination preparation questions.
• To contextualize AI performance against human physician benchmarks.

Main Methods:

• Analysis of 266 multiple-choice questions from the 2023 PREP Self-Assessment.
• Evaluation of DeepSeek-R1, ChatGPT-4, and ChatGPT-4.5.
• Comparison of AI performance against published American Board of Pediatrics first-time pass rates.

Main Results:

• DeepSeek-R1 achieved 98.1% accuracy, exceeding typical pass rates.
• ChatGPT-4.5 reached 96.6% accuracy, at the upper threshold of human performance.
• ChatGPT-4 demonstrated 82.7% accuracy, comparable to lower human pass rates.
• AI models struggled with integrating complex clinical presentations and rare disease knowledge.

Conclusions:

• DeepSeek-R1's exceptional performance suggests potential in medical education and clinical support.
• Further research is needed to explore AI's capabilities in complex clinical reasoning.