Correcting Performance Metrics Bias During Generalization from Biased Samples to Populations.

Area of Science:

• Biostatistics
• Machine Learning in Healthcare
• Epidemiology

Background:

• Prediction algorithm performance is crucial in healthcare but often evaluated on non-representative patient samples.
• Metrics like sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) can be inaccurate when applied to different populations than those sampled.
• Challenges arise when samples are deliberately biased or when inferring performance to a distinct patient population.

Purpose of the Study:

• To develop and illustrate methods for correcting prediction algorithm performance metrics (sensitivity, specificity, PPV, NPV).
• To address challenges of biased sampling and population inference for these metrics.
• To compare standard cell weighting and logistic regression weighting for performance metric correction.

Main Methods:

• Utilized inverse probability weighting methods, specifically standard cell weighting and logistic regression weighting.
• Derived corrected formulas for performance metrics based on underlying patient distributions.
• Conducted simulation experiments, including identifying patients with dementia, to compare correction methods across various sample sizes and prevalence settings.

Main Results:

• Inverse probability weighting methods effectively correct estimated algorithm performance metrics.
• Standard cell weighting demonstrated superior performance over logistic regression weighting in scenarios with small sample sizes and limited strata information.
• The study empirically validated the utility of weighting methods for improving metric accuracy.

Conclusions:

• Weighting methods provide a robust approach to adjust prediction algorithm performance metrics for biased samples and population inference.
• Standard cell weighting is a preferred method for performance correction under conditions of small sample size and available strata data.
• Accurate performance evaluation is essential for reliable clinical decision-making based on prediction algorithms.