Related Experiment Videos
Reproducibility of predictor variables from a validated clinical rule
P S Heckerling1, R C Conant, T G Tape
1Department of Medicine, University of Illinois, Chicago 60680.
Summary
Clinical prediction rules can be reproducible. Larger sample sizes (500) significantly improved the reproducibility of pneumonia prediction models compared to smaller sizes (250).
Area of Science:
- Medical Informatics
- Clinical Epidemiology
- Biostatistics
Background:
- Clinical prediction rules (CPRs) are essential for medical decision-making.
- Concerns exist regarding the reproducibility of CPRs and the stability of key variables in replicate models.
Purpose of the Study:
- To assess the reproducibility of a validated clinical prediction rule for radiographic pneumonia.
- To evaluate the impact of sample size on the reproducibility of CPRs.
Main Methods:
- Generated 200 replicate samples (sizes 250 and 500) from a training cohort of 905 patients.
- Applied stepwise logistic regression with forward selection among 31 variables.
- Assessed reproducibility using six criteria, including variable inclusion and order.
Main Results:
- Models from sample size 500 demonstrated higher reproducibility across all criteria compared to sample size 250 (e.g., 85.5% vs. 49.0% for including top 2 variables).
- Mean ROC areas were comparable between training and validation sets, with better agreement in larger sample sizes (80.5% for size 500 vs. 75.3% for size 250).
Conclusions:
- Larger sample sizes substantially enhance the reproducibility of clinical prediction rules.
- The validated pneumonia prediction rule shows good performance and reproducibility, particularly with adequate sample sizes.