Examining ChatGPT Performance on USMLE Sample Items and Implications for Assessment.

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Licensing Examinations

Background:

• Recent reports suggest ChatGPT possesses credible medical knowledge, evidenced by its potential to pass the United States Medical Licensing Examination (USMLE).
• The rapid advancement of AI necessitates evaluating its capabilities in high-stakes professional assessments.

Purpose of the Study:

• To analyze the generalizability of AI performance on sample USMLE items to actual examination performance.
• To assess ChatGPT's (version 3.5) accuracy and consistency on USMLE sample questions.

Main Methods:

• Analysis of publicly available USMLE sample items.
• ChatGPT (version 3.5) was queried three times per item to assess response stability.
• Responses were scored using operational scoring rules, with analysis of item characteristics and difficulty.

Main Results:

• ChatGPT consistently scored above 60% correct across most USMLE sample items, aligning with approximate passing standards for Steps 1 and 2.
• Response success varied across replications for 20% of items, and performance was weaker on practice-based learning questions.
• A modest correlation was observed between AI performance and item difficulty, with easier items answered more accurately.

Conclusions:

• ChatGPT's performance suggests potential consistency with passing standards for USMLE Steps 1 and 2, but limitations exist for Step 3 extrapolation.
• Variability in AI responses highlights the critical need for expert validation to ensure reliability and utility in medical education.
• The study underscores the importance of considering AI response stability and item-specific challenges in evaluating AI for medical licensing.