Gradient Tree Boosting for Hierarchical Data

Area of Science:

• Machine Learning
• Statistical Modeling
• Data Science

Background:

• Gradient tree boosting is effective for prediction but can struggle with hierarchical data.
• Ignoring hierarchical structures (e.g., longitudinal, clustered) can reduce predictive accuracy.
• Existing methods combine tree-based models with linear mixed-effects models (MEM) for hierarchical data.

Purpose of the Study:

• To propose and evaluate two novel algorithms combining MEM and gradient tree boosting.
• To assess the predictive performance of these new algorithms on hierarchical data.
• To compare MEM boosting against standard boosting, random forests, and other hierarchical data methods.

Main Methods:

• Development of two algorithms for estimating MEM gradient tree boosting.
• Simulation studies to investigate predictive performance.
• Comparison with standard gradient tree boosting, random forest, MEM, MEM random forests, model-based boosting, and Bayesian additive regression trees (BART).

Main Results:

• The proposed MEM boosting algorithms showed substantial predictive performance improvements over standard gradient tree boosting.
• MEM boosting and BART achieved predictive performance comparable to a correctly specified MEM.
• MEM boosting and BART generally outperformed model-based boosting and random forest approaches.

Conclusions:

• Combining MEM with gradient tree boosting effectively addresses hierarchical data structures.
• The proposed MEM boosting algorithms offer a powerful alternative for predictive modeling with hierarchical data.
• These methods provide competitive or superior performance compared to existing techniques.