Extensive benchmarking of a method that estimates external model performance from limited statistical...

Area of Science:

• Medical Informatics
• Health Services Research
• Biostatistics

Background:

• External validation is crucial for assessing predictive model generalizability.
• Access to external patient-level data for validation is often restricted.
• Existing methods for external validation can be data-intensive.

Purpose of the Study:

• To benchmark a novel method for estimating external model performance using only summary statistics.
• To evaluate the accuracy of this method across various metrics and datasets.
• To demonstrate the feasibility of assessing model transportability with limited external data.

Main Methods:

• The proposed method was benchmarked using five large, heterogeneous US data sources.
• Each data source served sequentially as the internal training set, with others acting as external validation sets.
• Performance was estimated using only external summary statistics.

Main Results:

• The method accurately estimated external model performance across multiple metrics.
• 95th percentile errors for discrimination, calibration, and overall accuracy were low (0.03, 0.08, 0.0002, and 0.07).
• These findings confirm the feasibility of estimating model transportability.

Conclusions:

• Estimating external model performance using summary statistics is feasible and accurate.
• This approach can accelerate the deployment of predictive models by simplifying external validation.
• The method offers a practical solution for assessing model generalizability with limited data access.