Bayesian Sample Size Calculations for External Validation Studies of Risk Prediction Models.

Area of Science:

• Biostatistics
• Epidemiology
• Health Informatics

Background:

• Current sample size calculations for external validation of risk prediction models rely on fixed performance metrics and precision targets.
• This approach is limited by uncertainty in true model performance due to finite previous study samples.
• Conventional methods are less suitable for assessing clinical utility using measures like net benefit (NB).

Purpose of the Study:

• To propose a general Bayesian framework for multi-criteria sample size considerations for prediction models with binary outcomes.
• To develop sample size rules for statistical performance metrics (discrimination, calibration) and clinical utility (net benefit).
• To address limitations of conventional methods by incorporating uncertainty and providing flexible sample size determination.

Main Methods:

• Developed a Bayesian framework for sample size calculations.
• Proposed sample size rules targeting expected precision or assurance probability for performance metrics.
• Introduced rules for net benefit based on Optimality Assurance and Value of Information (VoI) analysis.
• Applied the framework to validate a COVID-19 patient deterioration risk model.

Main Results:

• The Bayesian approach quantifies uncertainty in model performance, enabling flexible sample size rules.
• Value of Information (VoI) analysis for net benefit suggested lower sample sizes compared to precision-based calibration metrics.
• Demonstrated application in a case study for a COVID-19 risk prediction model.

Conclusions:

• A Bayesian framework offers a more comprehensive approach to sample size calculation for external model validation.
• This method allows for flexible sample size determination based on expected precision, assurance probabilities, and VoI.
• The proposed approach, particularly VoI for net benefit, can lead to more efficient sample size requirements.