Exploring the Association Between USMLE Scores and ACGME Milestone Ratings: A Validity Study Using National Data From...

Area of Science:

• Medical Education Assessment
• Graduate Medical Training Evaluation
• Physician Competency Measurement

Background:

• The United States Medical Licensing Examination (USMLE) and Accreditation Council for Graduate Medical Education (ACGME) milestones are key components of medical education assessment.
• Evaluating the association between standardized test scores and competency-based milestones is crucial for refining assessment strategies.

Purpose of the Study:

• To examine the relationship between USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores and ACGME Emergency Medicine (EM) milestone ratings.
• To provide validity evidence for the use of USMLE scores in assessing EM resident performance.

Main Methods:

• Subject matter experts (SMEs) identified expected associations between USMLE scores and EM subcompetencies.
• National-level data from 6,618 EM residents (2013-2018) were analyzed using bivariate correlations and multilevel regression.
• Empirical correlations were compared against SME judgments for convergent and discriminant validity.

Main Results:

• Step 2 CK scores demonstrated small but statistically significant positive relationships with most EM subcompetencies, aligning with SME predictions for 7 of 9 subcompetencies.
• A 1-SD increase in Step 2 CK score predicted a 0.12 increase in the MK-01 milestone rating, controlling for Step 1.
• Step 1 scores showed a statistically significant, albeit small, association only with the MK-01 subcompetency.

Conclusions:

• The findings offer incremental validity evidence supporting the use of USMLE Step 1 and Step 2 CK scores in conjunction with EM milestone ratings.
• These results reinforce the utility of standardized examinations in the continuum of medical education assessment.