Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection

Area of Science:

• Machine Learning
• Data Science
• Statistical Modeling

Background:

• Gradient Boosting Machines (GBM) are widely used for tabular data prediction.
• Standard GBM implementations often use decision trees biased towards high-cardinality categorical variables.
• This bias has been studied for predictive performance but not extensively for feature importance.

Purpose of the Study:

• To investigate the impact of biased base learners on feature importance (FI) in Gradient Boosting Machines.
• To propose and evaluate a method for mitigating this bias in GBM feature importance measures.

Main Methods:

• Utilizing cross-validated (CV) unbiased base learners within the GBM framework.
• Testing the proposed framework on both synthetic and real-world datasets.
• Comparing feature importance measures and predictive performance against standard GBM implementations.

Main Results:

• Biased base learners in GBM implementations lead to surprisingly biased feature importance, despite competitive predictive performance.
• The proposed framework using CV unbiased base learners significantly improves all GBM feature importance measures.
• Prediction accuracy is maintained at a comparable level to standard GBM implementations.

Conclusions:

• The bias in base learners significantly affects GBM feature importance, not just predictive performance.
• Employing cross-validated unbiased base learners is an effective and computationally efficient method to correct feature importance bias in GBM.
• This approach enhances the reliability of feature importance analysis in Gradient Boosting Machines.