Implementing machine learning methods with complex survey data: Lessons learned on the impacts of accounting sampling...

Area of Science:

• Epidemiologic research
• Machine learning
• Survey data analysis

Background:

• Machine learning methods are increasingly used in epidemiology, but software often lacks support for complex survey data.
• Analyzing complex survey data requires accounting for sampling weights for valid predictions in target populations.

Purpose of the Study:

• To determine if ignoring sampling weights in gradient boosting models affects prediction accuracy for all-cause mortality.
• To assess the impact of sample size, weight variability, predictor strength, and model dimensionality on prediction accuracy.

Main Methods:

• Utilized data from 15,820 participants in the National Health and Nutrition Examination Survey (1988-1994).
• Employed gradient boosting models to predict all-cause mortality, comparing weighted and unweighted analyses.
• Conducted simulations to evaluate various factors influencing prediction accuracy.

Main Results:

• Unweighted models in the NHANES data showed inflated F1 scores (81.9%) compared to weighted models (77.4%).
• Recalculating unweighted model performance using weighted outcomes mitigated this error (F1: 77.0%).
• In simulations, this mitigation effect was consistent for large sample sizes (N=10,000) but varied for smaller samples.

Conclusions:

• Failing to account for sampling weights in gradient boosting models can limit the generalizability of findings from complex survey data.
• Post-hoc recalculations of model performance using weighted observed outcomes can provide a more accurate prediction assessment when weighted algorithms are unavailable.