Artificial Intelligence Outperforms Physicians in General Medical Knowledge, Except in the Paediatrics Domain: A...

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Clinical Practice

Background:

• Generative artificial intelligence (genAI) demonstrates potential in clinical settings.
• Physician knowledge retention and domain-specific performance are critical for patient care.

Purpose of the Study:

• To compare the medical knowledge of a GPT-4-turbo virtual assistant against physicians from Italy, France, Spain, and Portugal.
• To analyze knowledge retention over time and performance across different medical domains.

Main Methods:

• A digital platform collected 221,574 answers from 17,144 physicians to 600 exam questions (December 2022 - February 2024).
• Physicians were stratified by years since graduation and specialty.
• Statistical analysis included binomial logistic regression and Fisher's exact test to compare assistant and physician performance.

Main Results:

• The GPT-4-turbo assistant achieved higher accuracy (72-96%) than physicians (46-62%) across all participating countries (p < 0.001).
• Physicians underperformed compared to the AI in most medical domains, with an exception in pediatrics (45% vs. 52%, p = 0.60).
• Physician accuracy decreased with seniority, showing a 4-10% decline in older cohorts (p < 0.001).

Conclusions:

• Generative AI surpasses practicing physicians in broad medical knowledge assessment.
• GenAI tools may offer a solution to mitigate knowledge attrition among medical professionals.
• Pediatrics represents a specific domain where further refinement of AI models is necessary.