Related Experiment Video
Updated: Jul 11, 2026
07:35
Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances
Published on: October 11, 2018
On the overestimation of random forest's out-of-bag error
Silke Janitza1, Roman Hornung1
1Institute for Medical Information Processing, Biometry and Epidemiology, University of Munich, Munich, Germany.
Plos One
|August 7, 2018
Summary
The out-of-bag error in random forests can overestimate prediction error for binary classification with metric predictors, especially in unbalanced or small datasets. Stratified subsampling is recommended for more accurate error estimation and parameter tuning.
Area of Science:
- Bioinformatics
- Machine Learning
- Statistical Modeling
Background:
- Random forests are popular for classification in bioinformatics.
- Out-of-bag error is a common technique for evaluating random forest accuracy and tuning parameters like mtry.
- Out-of-bag error can overestimate true prediction error in binary classification with metric predictors.
Purpose of the Study:
- Identify settings where out-of-bag error overestimation is likely for random forests.
- Investigate the reliability of out-of-bag error for tuning parameter selection (mtry).
Main Methods:
- Simulated and real data analysis.
- Evaluation of random forests with metric predictors.
- Comparison of out-of-bag error estimation with stratified subsampling.
Main Results:
- Overestimation of out-of-bag error is largest in balanced settings, with few observations, many predictors, low predictor correlation, and weak effects.
- Overestimation had minimal impact on tuning parameter selection in the studied cases.
- Stratified subsampling provided less biased estimates of true prediction error.
Conclusions:
- Out-of-bag error can be unreliable for error estimation and parameter tuning in random forests with metric predictors.
- Stratified subsampling (proportional to class sizes) is recommended for metric predictor variables.
- For unbalanced settings, sampling an equal number of observations per class is a viable alternative.
Related Concept Videos
03:37Uncertainty in Measurement: Accuracy and Precision
Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy of their results. Measurements are said to be precise if they yield very similar results when repeated in the same manner. A measurement is considered accurate if it yields a result that is very close to the true or the accepted value. Precise values agree with each other; accurate values agree with a true value.
01:20Random and Systematic Errors
Scientists always try their best to record measurements with the utmost accuracy and precision. However, sometimes errors do occur. These errors can be random or systematic. Random errors are observed due to the inconsistency or fluctuation in the measurement process, or variations in the quantity itself that is being measured. Such errors fluctuate from being greater than or less than the true value in repeated measurements. Consider a scientist measuring the length of an earthworm using a...
01:04Random Error
Random or indeterminate errors originate from various uncontrollable variables, such as variations in environmental conditions, instrument imperfections, or the inherent variability of the phenomena being measured. Usually, these errors cannot be predicted, estimated, or characterized because their direction and magnitude often vary in magnitude and direction even during consecutive measurements. As a result, they are difficult to eliminate. However, the aggregate effect of these errors can be...
00:59Propagation of Uncertainty from Random Error
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
01:20Random and Systematic Errors
Scientists always try their best to record measurements with the utmost accuracy and precision. However, sometimes errors do occur. These errors can be random or systematic. Random errors are observed due to the inconsistency or fluctuation in the measurement process, or variations in the quantity itself that is being measured. Such errors fluctuate from being greater than or less than the true value in repeated measurements. Consider a scientist measuring the length of an earthworm using a...
01:19Survival Tree
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a survival tree begins...
Building a Survival Tree
Constructing a survival tree begins...