Related Experiment Videos
Comparison and cross-validation of simple and multiple logistic regression models to predict USMLE step 1 performance
Anthony M Paolo1, Giulia A Bonaminio, Dianne Durham
1Office of Medical Education, University of Kansas School of Medicine, Kansas City, USA. apaolo@kumc.edu
Teaching and Learning in Medicine
|February 28, 2004
Summary
Cross-validating prediction models for the United States Medical Licensing Examination-Step 1 is crucial. The Year 2 Fall Grade Point Average (GPA) model showed the best generalizability for predicting student performance.
Area of Science:
- Medical Education Research
- Student Performance Prediction
- Licensing Examination Assessment
Background:
- Early identification of students at risk for United States Medical Licensing Examination-Step 1 (USMLE Step 1) difficulties is vital.
- Numerous predictive models exist, but few undergo rigorous cross-validation.
Purpose of the Study:
- To cross-validate existing prediction models for USMLE Step 1 performance.
- To assess the generalizability of different predictive models.
Main Methods:
- Development sample (n=686) and cross-validation sample (n=147) from a Midwestern medical school.
- Logistic regression for a multiple predictor model; Year 1 and Year 2 Fall Grade Point Average (GPA) as simple models.
- Receiver Operating Characteristic (ROC) curves, Kappa coefficients, sensitivity, and specificity were used for analysis.
Main Results:
- Year 1 GPA model showed poor agreement; other models demonstrated fair agreement with actual Step 1 performance.
- Multiple and Year 1 GPA models experienced significant accuracy loss upon cross-validation.
- The Year 2 Fall GPA model maintained classification accuracy during cross-validation.
Conclusions:
- Cross-validation is essential for evaluating the generalizability and practical utility of USMLE Step 1 prediction models.
- The Year 2 Fall GPA model appears more robust for predicting Step 1 performance across different student cohorts.