Resampling methods improve the predictive power of modeling in class-imbalanced datasets

Area of Science:

• Medical Informatics
• Machine Learning in Healthcare
• Public Health Data Analysis

Background:

• Dichotomized medical outcome variables often result in class-imbalanced datasets, where one class significantly outnumbers the other.
• Standard classification models perform suboptimally on imbalanced datasets, potentially leading to biased predictions in clinical settings.
• Resampling techniques, including oversampling and undersampling, are proposed strategies to mitigate the challenges posed by class imbalance.

Purpose of the Study:

• To illustrate the impact of resampling methods on classification model performance using a real-world health dataset.
• To evaluate the effectiveness of oversampling and undersampling in improving the prediction of undiagnosed diabetes.
• To assess the influence of resampling on various extensions of the Classification and Regression Tree (CART) algorithm.

Main Methods:

• Utilized the National Health and Nutrition Examination Survey (NHANES) 2009-2010 dataset, analyzing 4677 participants without self-reported diabetes.
• Developed Classification and Regression Tree (CART) models to predict diabetes based on WHO criteria, using demographics and socio-economic status as predictors.
• Applied oversampling, undersampling, and weighted sampling techniques to the training data, varying case-to-control ratios, and evaluated model performance using Area Under the ROC Curve (AUC) on a separate testing set.

Main Results:

• CART models trained on oversampled (AUC = 0.70) and undersampled (AUC = 0.74) data demonstrated superior classification power compared to models trained on the original imbalanced data (AUC = 0.65).
• Resampling methods also enhanced the performance of other CART-based algorithms, including random forests and generalized boosted trees.
• Different resampling ratios (1:1, 1:2, 1:4) were examined, indicating a general improvement in predictive accuracy across various configurations.

Conclusions:

• Applying resampling methods is an effective strategy to improve the classification power of machine learning models on class-imbalanced medical datasets.
• Oversampling and undersampling techniques provide significant performance gains for CART, random forests, and generalized boosted trees in predicting conditions like diabetes.
• The findings underscore the importance of addressing class imbalance in health data analysis to achieve more reliable and accurate predictive models.