Improving logistic regression on the imbalanced data by a novel penalized log-likelihood function

Area of Science:

• Machine Learning
• Statistical Modeling

Background:

• Logistic regression models often exhibit bias on imbalanced datasets, favoring the majority class and leading to practical losses.
• Current bias mitigation strategies for logistic regression involve complex hyperparameter tuning or high computational costs.

Purpose of the Study:

• To propose a novel penalized log-likelihood function for logistic regression that effectively handles imbalanced data.
• To improve the discrimination ability and computational efficiency of logistic regression models.

Main Methods:

• Developed a penalized log-likelihood function incorporating learnable penalty weights for minority class observations.
• Integrated these penalty weights as decision variables learned alongside model coefficients.
• Evaluated the proposed model against existing methods using Area Under the Receiver Operating Characteristics (ROC) curve on public and simulated datasets.

Main Results:

• The proposed logistic regression model demonstrated improved discrimination ability (ROC curve) and enhanced computational efficiency compared to existing methods.
• Analysis on an imbalanced credit dataset showed better performance in terms of type I and type II errors.
• The learned penalty weights effectively adjusted for class imbalance without requiring manual hyperparameter estimation.

Conclusions:

• The novel penalized log-likelihood function offers a more effective and efficient approach for logistic regression on imbalanced data.
• This method enhances model performance and reduces bias towards the majority class.
• The approach provides a valuable alternative for applications dealing with skewed datasets.