Predicting Step 2 CK Performance Using Automated Feature Selection and Nested Cross-Validation

Area of Science:

• Medical Education
• Biostatistics
• Health Informatics

Background:

• The USMLE Step 1 exam's shift to pass/fail necessitates advanced prediction models for the USMLE Step 2 Clinical Knowledge (USMLE-CK) exam.
• Accurate prediction of USMLE-CK performance is vital for evaluating medical student competency.

Purpose of the Study:

• To develop and validate a predictive model for USMLE-CK scores using multiple linear regression and automated feature selection.
• To provide medical educators with a tool to forecast student performance on the USMLE-CK exam.

Main Methods:

• A nested cross-validation framework was employed for feature selection and model validation.
• Data from four undergraduate medical student cohorts (n=117) were analyzed, including internal assessments and National Board Medical Examination (NBME) Clinical Science Subject Exams scores.
• Multiple regression models were evaluated using metrics such as mean CV error, adjusted-R², Mallows' Cp, and Bayesian Information Criteria.

Main Results:

• A four-predictor model was selected, achieving an adjusted-R² of 0.68.
• The optimal model incorporated scores from NBME exams in Medicine, Neurology, Surgery, and performance in a pre-clinical Gastrointestinal unit.

Conclusions:

• The study successfully merged feature selection and model validation to create a streamlined predictive modeling process.
• An interactive dashboard was developed to empower medical educators in predicting USMLE-CK performance.
• This approach shows potential for predicting student performance in other high-stakes assessments.