Related Experiment Video
Updated: May 27, 2026
12:51
Evaluation of Commercial-Off-The-Shelf Wrist Wearables to Estimate Stress on Students
Published on: June 16, 2018
A Machine Learning Approach to Identifying Students at Risk for USMLE Step 1 Exam Delay Using Academic Performance
Min-Jung Kim1,2,3, Zita Lazzarini4, Thomas M Manger5
1Calhoun Cardiology Center, School of Medicine, University of Connecticut, Farmington, CT, United States.
Summary
Machine learning models can predict delays in the United States Medical Licensing Examination (USMLE) Step 1 exam. Early academic indicators like self-assessments and course scores identify students at risk, enabling timely interventions.
Area of Science:
- Medical education research
- Machine learning in healthcare
- Student performance analysis
Background:
- Delaying the United States Medical Licensing Examination (USMLE) Step 1 exam is associated with lower academic performance.
- Predicting which students may delay their Step 1 exam is crucial for timely academic support.
Purpose of the Study:
- To develop and evaluate explainable machine learning (XML) models for predicting USMLE Step 1 exam delays.
- To identify key predictors associated with the risk of delaying the Step 1 exam.
Main Methods:
- Analyzed data from 610 medical students, with 27.4% delaying their Step 1 exam.
- Evaluated XML models (RF, XGBoost, CatBoost, logistic regression) using cross-validation.
- Utilized SHAP and permutation feature importance for model interpretability.
Main Results:
- XGBoost demonstrated superior predictive performance (AUROC: 0.80).
- Key predictors included incorrect responses on the Comprehensive Basic Science Self-Assessment (CBSSA) and preclerkship course scores.
- Students who delayed the exam had significantly more incorrect CBSSA responses and lower course scores.
Conclusions:
- Ensemble machine learning, especially XGBoost, effectively predicts students at risk of delaying the USMLE Step 1 exam.
- Early identification of at-risk students using academic indicators can facilitate interventions to support academic progression.