Evaluating the three-level approach of the U-smile method for imbalanced binary classification

Area of Science:

• Machine Learning
• Statistical Modeling
• Medical Informatics

Background:

• Real-world binary classification frequently uses imbalanced datasets, posing challenges for model evaluation.
• The U-smile method was previously developed and validated for assessing variable usefulness under class balance.

Purpose of the Study:

• To evaluate the U-smile method's performance under class imbalance.
• To propose a three-level approach for the U-smile method incorporating I coefficients for weighting point size.

Main Methods:

• The U-smile method (U-smile plot, BA, RB, and I coefficients) was evaluated on the Heart Disease dataset and generated data.
• Logistic regression models were built to assess four new variables at seven imbalance levels (1% to 99%).
• U-smile results were compared against traditional measures like Brier skill score, net reclassification index, and AUC difference.

Main Results:

• The reference model exhibited overfitting to the majority class at higher imbalance levels.
• The BA-RB-I coefficients successfully identified informative variables across all imbalance levels.
• The U-smile method indicated improved minority class prediction (positive BA, I) and reduced majority class overfitting (negative RB) at higher imbalances.
• U-smile outperformed traditional measures in variable selection for imbalanced binary classification.

Conclusions:

• The U-smile method is a robust tool for variable selection in imbalanced binary classification.
• Its ability to handle class imbalance makes it valuable for prevalent real-life scenarios.