Evaluating Binary Outcome Classifiers Estimated from Survey Data

Area of Science:

• Epidemiology
• Health Sciences
• Social and Behavioral Sciences

Background:

• Surveys are vital research tools but often use complex sampling designs, not simple random samples.
• Survey respondents are typically assigned weights to account for unequal selection probabilities.
• Evaluating predictive models on survey data requires careful consideration of these complex designs.

Purpose of the Study:

• To demonstrate the benefit of using survey weights for assessing predictive model quality.
• To compare weighted versus unweighted performance metrics on complex survey data.
• To evaluate the impact of weighting on models trained with class imbalance mitigation.

Main Methods:

• Characterized model assessment statistics (e.g., sensitivity, specificity) as finite population quantities.
• Computed survey-weighted estimates using random subsets of original survey data for testing.
• Conducted simulations using data from the National Survey on Drug Use and Health and National Comorbidity Survey.

Main Results:

• Unweighted metrics using sample test data can inaccurately represent population performance.
• Weighted metrics appropriately adjust for complex sampling designs, providing accurate population estimates.
• The benefit of weighted metrics persists even when models are trained using upsampling for class imbalance.

Conclusions:

• Survey weights are crucial for accurate predictive model performance evaluation on complex survey data.
• Weighted metrics provide a more reliable assessment of model generalizability to the target population.
• Researchers should adopt weighted metrics when evaluating models trained or tested on complex survey datasets.