Estimating prediction error for complex samples

Area of Science:

• Statistics
• Biostatistics
• Machine Learning

Background:

• Assessing prediction model utility requires accounting for non-representative training samples.
• Estimating generalization error for models trained on complex survey data is challenging.
• Existing methods like Efron's estimator assume representative data.

Purpose of the Study:

• To extend Efron's prediction error estimation to complex survey data.
• To develop a consistent estimator for generalization error in non-uniform samples.
• To provide a more widely applicable alternative to dAIC for survey data.

Main Methods:

• Incorporation of Horvitz-Thompson sampling weights into Efron's covariance penalty estimator.
• Development of the Horvitz-Thompson-Efron estimator.
• Validation through simulations and application to renal function prediction models.

Main Results:

• The Horvitz-Thompson-Efron estimator is consistent for the true generalization error rate.
• The proposed estimator is equivalent to dAIC but more broadly applicable.
• Demonstrated utility in predicting renal function from NHANES survey data.

Conclusions:

• The Horvitz-Thompson-Efron estimator effectively addresses generalization error in complex survey data.
• This method enhances the reliability of prediction models trained on non-representative samples.
• The approach has practical implications for health outcome prediction using large-scale surveys.