Contemporary Step 1 Predictive Methods Across 12 US MD-Granting Medical Schools

Area of Science:

• Medical Education
• Assessment and Evaluation
• Educational Data Mining

Background:

• The United States Medical Licensing Examination (USMLE) Step 1 scoring shifted to pass/fail in 2022, leading to increased national first-time failure rates.
• This change necessitates improved, data-driven methods for identifying medical students at risk of failing Step 1 to enable timely interventions.
• Medical school structures and assessment methods vary, complicating the development of universally applicable risk identification models.

Purpose of the Study:

• To compare the predictive models used by 12 US MD-granting medical schools to identify students at risk of failing the USMLE Step 1 exam.
• To assess the sensitivity, specificity, and timing of risk identification across different predictive methodologies.
• To identify key features and provide recommendations for developing, implementing, and refining Step 1 predictive models.

Main Methods:

• A comparative analysis of 12 US medical schools' USMLE Step 1 predictive models.
• Data collection involved virtual meetings and surveys detailing each institution's risk identification methods.
• Models were evaluated based on their ability to predict Step 1 failure, including sensitivity, specificity, and the timing of risk identification relative to the exam date.

Main Results:

• Six institutions used categorical risk assessment, three used multiple regression, two combined approaches, and one used growth mixture modeling.
• Performance on National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination (CBSE) or Comprehensive Basic Science Self-Assessment (CBSSA) exams were the most significant predictors across institutions.
• Most schools identified at-risk students at the beginning of their dedicated Step 1 study period, with varying intervention thresholds.

Conclusions:

• Contemporary USMLE Step 1 predictive models exhibit diverse methodologies with inherent strengths and limitations.
• While NBME exams are strong predictors, the effectiveness of interventions varies due to differing school thresholds.
• Continuous improvement of predictive model performance is crucial for enhancing student learning outcomes and success on the USMLE Step 1.