Sample size calculation to externally validate scoring systems based on logistic regression models

Area of Science:

• Biostatistics
• Clinical Epidemiology
• Health Informatics

Background:

• Traditional recommendations suggest at least 100 events and 100 non-events for predictive model validation.
• Factors like discrimination, parameterization, and incidence can influence predictive model calibration.
• Scoring systems derived from binary logistic regression models are common predictive tools.

Purpose of the Study:

• To develop and present an algorithm for calculating the optimal sample size required for validating scoring systems based on binary logistic regression.
• To apply this novel algorithm to a real-world case study for practical demonstration.

Main Methods:

• The algorithm utilizes bootstrap sampling to assess key validation metrics.
• It calculates the area under the ROC curve (AUC) for discrimination.
• It evaluates observed event probabilities via smooth curves and the estimated calibration index (ECI) for calibration assessment.

Main Results:

• The case study application demonstrated the algorithm's efficacy.
• The algorithm determined an adequate sample size requiring only 69 events.
• This sample size is notably lower than the commonly cited benchmark of 100 events.

Conclusions:

• An effective algorithm is presented for determining the sample size needed to validate binary logistic regression-based scoring systems.
• This methodology offers a more precise approach to sample size calculation for model validation.
• The algorithm's applicability extends to similar validation scenarios in various research contexts.