A comparison of random forest variable selection methods for regression modeling of continuous outcomes

Area of Science:

• Machine Learning
• Statistical Modeling
• Data Science

Background:

• Random Forest (RF) regression is a widely used machine learning technique for developing prediction models with continuous outcomes.
• Variable selection, or feature selection, is crucial for enhancing model accuracy, computational efficiency, and data collection practicality.
• Limited evidence exists to guide the selection of optimal RF variable selection methods across diverse datasets.

Purpose of the Study:

• To benchmark and compare the performance of 13 random forest variable selection methods for continuous outcomes.
• To provide evidence-based guidance on selecting preferable variable selection methods for different dataset types.
• To assess variable selection methods using an open science approach.

Main Methods:

• A benchmarking study evaluated 13 RF variable selection methods using 59 publicly available datasets.
• Performance was measured by out-of-sample R2 of RF models built with selected variables.
• Efficiency was assessed by computational time and simplicity by the percentage of variables reduced.

Main Results:

• Variable selection methods in the Boruta and aorsf R packages performed best for axis-based RF models.
• Methods within the aorsf R package demonstrated superior performance for oblique RF models.
• The study identified specific RF variable selection methods suitable for different RF model types.

Conclusions:

• The Boruta and aorsf R packages offer effective variable selection for axis-based RF regression.
• The aorsf R package is recommended for variable selection in oblique RF regression models.
• This research provides valuable insights for optimizing RF regression models through informed variable selection.