Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of...

Area of Science:

• Statistics
• Biostatistics
• Clinical Epidemiology

Background:

• Minimum sample size formulas (e.g., Riley et al.) aim to prevent overfitting in clinical prediction model development.
• While effective on average, the variability of overfitting at recommended sample sizes remains unclear.

Purpose of the Study:

• To investigate the variability of overfitting in logistic regression clinical prediction models using simulation and empirical data.
• To compare unpenalised maximum likelihood estimation with post-estimation shrinkage or penalisation methods.

Main Methods:

• Simulation study and empirical example.
• Development of logistic regression models using unpenalised maximum likelihood estimation.
• Application of various post-estimation shrinkage or penalisation techniques.

Main Results:

• Mean calibration slopes were near one for all methods, indicating good average calibration.
• Penalisation methods reduced average overfitting compared to unpenalised methods.
• Penalisation methods exhibited higher variability in external predictive performance.

Conclusions:

• Penalisation methods are recommended alongside minimum sample size requirements to further mitigate overfitting in clinical prediction models.
• Examining variability in predictive performance and tuning parameters is crucial for robust model development and assures performance in new individuals.