Related Experiment Videos
Internal validation of predictive models: efficiency of some procedures for logistic regression analysis
E W Steyerberg1, F E Harrell, G J Borsboom
1Center for Clinical Decision Sciences, Ee 2091, Department of Public Health, Erasmus University, P.O. Box 1738, 3000 DR, Rotterdam, The Netherlands. steyerberg@mgz.fgg.eur.nl
Journal of Clinical Epidemiology
|July 27, 2001
Summary
Estimating predictive model performance requires internal validation. Bootstrapping offers the most stable and accurate assessment for logistic regression models, outperforming split-sample and cross-validation methods.
Area of Science:
- Medical Statistics
- Clinical Epidemiology
- Health Informatics
Background:
- Predictive model performance is often overestimated when validated on the same data used for development.
- Internal validation methods are crucial for estimating how a model will perform on new, unseen data.
- Accurate performance estimation is vital for reliable clinical decision-making.
Purpose of the Study:
- To evaluate and compare the performance of different internal validation techniques for predictive logistic regression models.
- To determine the most reliable method for estimating model performance in new subjects.
- To assess the efficiency and accuracy of split-sample, cross-validation, and bootstrapping methods.
Main Methods:
- Evaluated variants of split-sample, cross-validation, and bootstrapping.
- Used a logistic regression model with eight predictors for 30-day mortality after acute myocardial infarction.
- Sampled data from the GUSTO-I dataset (n=40,830) to simulate various data set sizes and event-per-variable ratios.
Main Results:
- Split-sample validation yielded overly pessimistic and highly variable performance estimates.
- 10% cross-validation showed low bias and variability but was not universally applicable to all performance measures.
- Bootstrapping provided stable, low-bias estimates, offering the best internal validity assessment.
Conclusions:
- Split-sample validation is inefficient for assessing predictive model performance.
- Bootstrapping is the recommended method for estimating the internal validity of predictive logistic regression models.
- Accurate internal validation is essential for the reliable application of predictive models in clinical practice.