Improving random forest predictions in small datasets from two-phase sampling designs.

Area of Science:

• Machine Learning
• Biostatistics
• Epidemiology

Background:

• Random forests are powerful machine learning tools but require optimization for specific data structures.
• Biomedical studies often involve two-phase sampling with rare outcomes and resource-intensive covariate measurements.
• Optimizing random forest performance is crucial for these challenging datasets.

Purpose of the Study:

• To optimize random forest prediction performance for small datasets from two-phase sampling designs.
• To evaluate the impact of variable screening, class balancing, weighting, and hyperparameter tuning.
• To compare random forests with generalized linear models and explore ensemble methods.

Main Methods:

• Utilized an immunologic marker dataset from an HIV vaccine efficacy trial.
• Applied combinations of variable screening, class balancing, weighting, and hyperparameter tuning.
• Employed stacking of random forests and generalized linear models.

Main Results:

• Class balancing improved performance without variable screening but harmed it with screening.
• Weighting's impact depended on variable screening.
• Hyperparameter tuning was ineffective for small sample sizes.
• Random forests under-performed generalized linear models on some marker subsets.
• Stacking models improved prediction performance, dependent on learner prediction dissimilarities.

Conclusions:

• Variable screening and inverse sampling probability weighting are key for random forest performance in small, two-phase sampled datasets.
• Stacking random forests with linear models offers performance improvements.
• Careful method selection is vital for rare outcome prediction in biomedical research.