Protocol to benchmark and evaluate the status of imbalance measure using correlation, data complexity, and ablation...

Area of Science:

• Computer Science
• Data Science
• Machine Learning

Background:

• Machine learning algorithms often face challenges with imbalanced datasets, where one class significantly outnumbers others.
• Existing mitigation strategies' effectiveness is contingent on the degree of data imbalance.
• A standardized method to accurately assess data imbalance is needed to guide the selection of appropriate machine learning techniques.

Purpose of the Study:

• To develop and validate a comprehensive protocol for benchmarking various data imbalance measures.
• To evaluate the performance of different imbalance measures across diverse datasets and classifiers.
• To identify the most effective measure for quantifying data imbalance in machine learning.

Main Methods:

• A protocol was developed and applied to 428 synthetic and 70 real-world datasets.
• Eight distinct imbalance measures were benchmarked using multiple machine learning classifiers and evaluation metrics.
• Correlation coefficients and complexity analyses were employed to assess measure performance and efficiency.

Main Results:

• The study systematically evaluated multiple imbalance measures, providing a comparative analysis of their effectiveness.
• The developed protocol facilitates the determination of the extent of data imbalance.
• The SIMBA (status of imbalance) measure was identified as a highly efficient method through ablation studies.

Conclusions:

• The proposed protocol offers a robust framework for assessing data imbalance in machine learning.
• Accurate quantification of imbalance is essential for selecting appropriate mitigation strategies and improving model performance.
• The SIMBA measure presents a promising tool for addressing challenges posed by imbalanced datasets.