An empirical evaluation of sampling methods for the classification of imbalanced data

Area of Science:

• Machine Learning
• Data Science
• Statistical Modeling

Background:

• Imbalanced classification is common in fields like disease diagnosis and fraud detection.
• Standard machine learning methods struggle with imbalanced datasets.
• Data balancing via oversampling or undersampling is a popular but unevaluated solution.

Purpose of the Study:

• To rigorously evaluate the effectiveness of sampling methods in imbalanced classification.
• To assess combinations of sampling techniques with various machine learning classifiers.
• To analyze the impact of sampling on performance metrics like AUPRC and AUROC.

Main Methods:

• Evaluated seven sampling methods combined with eight machine learning classifiers (56 varieties).
• Utilized 31 datasets with varying degrees of class imbalance.
• Measured performance using areas under the precision-recall curve (AUPRC) and receiver operating characteristics curve (AUROC).

Main Results:

• Sampling significantly altered classifier performance in only a small percentage of cases (12.2% for AUPRC, 10.0% for AUROC).
• Sampling was more likely to reduce performance than improve it, particularly impacting AUPRC.
• Undersampling performed poorly, and sampling was more beneficial for linear classifiers.

Conclusions:

• The effectiveness of sampling for imbalanced classification is limited and can be detrimental.
• The choice of performance metric (AUPRC vs. AUROC) is critical for evaluating sampling strategies.
• Optimal classifiers were often achievable without sampling, challenging its widespread application.