Comparing variable selection and model averaging methods for logistic regression

Area of Science:

• Statistics
• Machine Learning
• Econometrics

Background:

• Model uncertainty is a significant challenge in statistical modeling, particularly for binary outcomes like logistic regression.
• The selection of appropriate predictors significantly impacts model performance and interpretation.
• Existing methods for addressing model uncertainty in logistic regression lack comprehensive comparative performance data under realistic conditions.

Purpose of the Study:

• To conduct a rigorous, simulation-based comparison of 28 established variable selection and inference methods for logistic regression.
• To evaluate method performance across diverse empirical datasets with varying sample sizes and predictor numbers.
• To identify the most effective strategies for handling model uncertainty in logistic regression, especially in the presence of data separation.

Main Methods:

• A preregistered, simulation-based study comparing 28 methods for variable selection and inference.
• Utilized 11 empirical datasets representing a range of sample sizes and predictor counts.
• Assessed performance in scenarios both with and without data separation (a common issue in binary outcome modeling).

Main Results:

• Bayesian model averaging (BMA) methods employing specific priors demonstrated superior performance when data separation was absent.
• Penalized likelihood approaches, notably the LASSO (Least Absolute Shrinkage and Selection Operator), yielded the most stable results in the presence of separation.
• BMA utilizing the empirical Bayes local (EB-local) prior proved competitive across both separated and non-separated datasets.

Conclusions:

• BMA with specific priors offers strong performance for logistic regression when data separation is not an issue.
• Penalized likelihood methods, particularly LASSO, are recommended for stable variable selection in logistic regression with data separation.
• The EB-local prior for BMA provides a robust alternative, performing well in both separated and non-separated scenarios, offering practical guidance for applied researchers.