Bayesian Ensemble Trees (BET) for Clustering and Prediction in Heterogeneous Data

Area of Science:

• Machine Learning
• Statistical Modeling
• Ensemble Methods

Background:

• Ensemble methods enhance predictive accuracy by combining multiple models.
• Classification and Regression Trees (CART) are widely used but can overfit.
• Existing ensemble techniques often require numerous trees, increasing computational cost.

Purpose of the Study:

• To propose a novel "tree-averaging" model called Bayesian Ensemble Trees (BET).
• To demonstrate BET's ability to adaptively determine the optimal number of trees.
• To showcase BET's efficiency and interpretability compared to other ensemble methods.

Main Methods:

• Utilizing an ensemble of Classification and Regression Trees (CART).
• Grouping data subsets and modeling them as a Dirichlet process (Bayesian Ensemble Trees).
• Developing an efficient estimation procedure for CART and mixture models.

Main Results:

• BET adapts to data heterogeneity, determining the optimal number of trees.
• BET achieves equivalent prediction accuracy with significantly fewer trees than other ensemble methods.
• Individual trees within BET provide variable selection criteria and subset interpretation.

Conclusions:

• BET offers a computationally efficient and interpretable alternative for ensemble modeling.
• The Dirichlet process modeling enables adaptive tree selection based on data characteristics.
• BET demonstrates strong performance in both simulated data and real-world regression tasks, such as predicting lung function.