Cross-Validated Variable Selection in Tree-Based Methods Improves Predictive Performance.

Area of Science:

• Machine Learning
• Data Science
• Statistical Modeling

Background:

• Recursive partitioning methods are standard in predictive modeling.
• Existing tree-building methods struggle with categorical variables, especially those with many categories.
• This limitation hinders the effective use of big data.

Purpose of the Study:

• To propose a novel framework for splitting variables in tree-based models.
• To enable the effective utilization of categorical variables with numerous categories.
• To improve the performance of single tree models and ensemble methods.

Main Methods:

• A framework using leave-one-out (LOO) cross-validation (CV) for splitting variable selection.
• Integration with existing splitting approaches like CART.
• Development of an efficient algorithm for LOO splitting variable selection.

Main Results:

• Categorical variables with many categories can be safely and effectively used.
• The proposed method improves predictive power by selecting variables that contribute to it.
• Significant performance enhancements observed in both single tree models and ensemble methods.
• Computational complexity remains comparable to CART for two-class classification.

Conclusions:

• The proposed LOO cross-validation splitting framework addresses a key limitation in tree-based modeling.
• This approach enhances the utility of high-cardinality categorical variables in big data scenarios.
• The method offers improved predictive accuracy and efficient computation for classification tasks.