Substantial effective sample sizes were required for external validation studies of predictive logistic regression...

Area of Science:

• Biostatistics
• Clinical Prediction Models
• Health Research Methodology

Background:

• Prediction model performance often declines in external validation datasets compared to development data.
• Assessing the reliability of prediction models in new populations is crucial for clinical utility.
• Determining optimal sample sizes for external validation is essential for robust model evaluation.

Purpose of the Study:

• To identify effective sample sizes, specifically the number of events, needed to detect meaningful differences in prediction model performance during external validation.
• To establish power requirements for detecting specific types of model invalidity.

Main Methods:

• Utilized logistic regression to predict benign tissue in residual masses for metastatic testicular cancer patients.
• Employed standard power calculations and Monte Carlo simulations to estimate required event numbers.
• Calculated sample sizes to detect specific model performance degradations with 80% power at a 5% significance level.

Main Results:

• Detecting overly optimistic probability predictions (1.5x odds scale) required 111 events in the validation sample.
• A decrease in discriminative ability (c-statistic from 0.83 to 0.73) necessitated 81 to 106 events, varying by scenario.

Conclusions:

• A minimum of 100 events and 100 nonevents is recommended for external validation samples.
• Substantially larger effective sample sizes may be necessary for specific hypothesis testing to achieve adequate statistical power.