Diversity Forests: Using Split Sampling to Enable Innovative Complex Split Procedures in Random Forests

Area of Science:

• Machine Learning
• Computational Statistics
• Data Mining

Background:

• Conventional random forests use univariable, binary splitting for predictive performance.
• Complex split procedures are needed to address issues like feature interactions.
• The diversity forest algorithm offers a new sampling scheme for complex splits.

Purpose of the Study:

• To introduce and evaluate the diversity forest algorithm, a novel candidate node split sampling scheme.
• To enable innovative complex split procedures within the random forest framework.
• To assess the computational feasibility and overfitting avoidance of complex splits.

Main Methods:

• The diversity forest algorithm samples splits from a candidate split set.
• This involves sampling a split problem and then sampling splits from that problem.
• Empirical evaluation used 220 datasets with binary outcomes, comparing diversity forests with conventional random forests and extremely randomized trees.

Main Results:

• The diversity forest split sampling scheme does not negatively impact random forest predictive performance.
• Performance remains robust across different values of the 'nsplits' parameter.
• Interaction forests, a type of diversity forest, effectively model and detect feature interactions.

Conclusions:

• Diversity forests provide a computationally tangible and overfitting-avoiding method for complex split procedures in random forests.
• This approach enhances the capability of random forests, particularly for tasks like interaction detection.
• The method shows promise for future development of advanced machine learning models.