Class imbalance should not throw you off balance: Choosing the right classifiers and performance metrics for brain...

Area of Science:

• Neuroscience
• Machine Learning
• Data Science

Background:

• Machine learning (ML) is vital in cognitive, computational, and clinical neuroscience.
• Imbalanced datasets are a common challenge in ML, potentially leading to severe consequences if unaddressed.

Purpose of the Study:

• To provide a didactic assessment of the class imbalance problem in neuroscience ML.
• To illustrate the impact of data imbalance on ML model performance using simulated and real brain data (EEG, MEG, fMRI).

Main Methods:

• Systematic manipulation of data imbalance ratios in simulated and brain imaging datasets.
• Evaluation of classification metrics, including Accuracy (Acc), Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC), and Balanced Accuracy (BAcc).
• Assessment of Random Forest (RF) robustness, stratified cross-validation, and hyperparameter optimization.

Main Results:

• The standard Accuracy (Acc) metric yields misleadingly high performances on imbalanced datasets by disregarding minority class performance.
• Balanced Accuracy (BAcc) and AUC provide more reliable performance evaluations for imbalanced data.
• Random Forest (RF) demonstrates robustness, and stratified cross-validation/hyperparameter optimization aid in addressing data imbalance.

Conclusions:

• For neuroscience ML applications aiming to minimize classification error, Balanced Accuracy (BAcc) is recommended for reliable performance evaluation.
• BAcc is equivalent to standard Acc on balanced data and extends to multi-class settings.
• Recommendations and open-source code are provided for the neuroscience community to address imbalanced data.