Related Experiment Video
Updated: May 27, 2026
Evaluation of Commercial-Off-The-Shelf Wrist Wearables to Estimate Stress on Students
Published on: June 16, 2018
A Machine Learning Approach to Identifying Students at Risk for USMLE Step 1 Exam Delay Using Academic Performance
Min-Jung Kim1,2,3, Zita Lazzarini4, Thomas M Manger5
1Calhoun Cardiology Center, School of Medicine, University of Connecticut, Farmington, CT, United States.
Purpose:
Delaying the United States Medical Licensing Examination (USMLE) Step 1 exam is often linked to poorer performance on Step 1, clerkship, and Step 2 Clinical Knowledge exams. This study developed and evaluated explainable machine learning (XML) models to predict Step 1 exam delay using student performance data and identify key predictors contributing to delay risk.
Method:
Data from 610 medical students at University of Connecticut School of Medicine for matriculation years 2016, 2017, 2019, 2020, 2021, and 2022 were analyzed, with 167 students (27.4%) delaying their Step 1 exam. XML models, including RF (random forest), XGBoost (Extreme Gradient Boosting), CatBoost (Categorical Boosting), and regularized logistic regression, were evaluated using 5-fold cross-validation and testing datasets. Misclassification rate (MCR), area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve (AUPRC) were used to compare model performance. Shapley Additive Explanations (SHAP) and permutation feature importance were applied to interpret key predictors.
Results:
XGBoost outperformed all other models, achieving a median (IQR) MCR of 0.14 (0.11-0.17), AUROC of 0.80 (0.76-0.84), and AUPRC of 0.72 (0.60-0.78). SHAP and permutation feature importance analyses revealed that incorrect responses on the first Comprehensive Basic Science Self-Assessment (CBSSA), along with preclerkship foundational basic science and laboratory course scores, were the most influential predictors of Step 1 exam delay. Students with delays had a median (IQR) of 99 (82-112) incorrect responses on the first CBSSA vs 75 (56-92) for nondelayers (P < .001). Students with delays had significantly lower scores across the 5 preclerkship course blocks.
Conclusions:
Ensemble machine learning models, particularly XGBoost, provide strong predictive performance for identifying students at risk for delaying the Step 1 exam. Early academic indicators, such as CBSSA and course performance, can inform timely interventions, supporting students' academic progression and success.