Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study.

Area of Science:

• Biostatistics
• Epidemiology
• Health Services Research

Background:

• Multicenter datasets are crucial for developing and validating clinical risk prediction models.
• Evaluating model performance across different centers and at the population level presents unique challenges.
• Standard statistical approaches may not fully capture the complexities of multicenter data structures.

Purpose of the Study:

• To present a comprehensive framework for evaluating the predictive performance of clinical risk models at both center and population levels.
• To investigate the impact of model choice, prediction type, and validation level on predictive performance.
• To provide recommendations for model development and validation in multicenter settings.

Main Methods:

• A simulation study was conducted to assess predictive performance under various conditions.
• Evaluated standard versus mixed-effects logistic regression models.
• Compared population-averaged, center-specific, and average random effect predictions.
• Assessed performance at both center and population validation levels.

Main Results:

• Calibration slopes deviate from one due to overfitting, model choice, prediction type, and validation level.
• Center-specific predictions demonstrated the best predictive performance in both center and population levels when data exhibited high clustering (ICC ≥ 20%).
• Model choice and prediction type significantly influenced performance metrics.

Conclusions:

• Clinical risk prediction models should be developed to reflect the inherent data structure of multicenter studies.
• The level of model validation should be aligned with the specific research question being addressed.
• Center-specific predictions are recommended for highly clustered multicenter data to enhance predictive accuracy.