Using Background Knowledge from Preceding Studies for Building a Random Forest Prediction Model: A Plasmode...

Area of Science:

• Medical Informatics
• Statistical Modeling
• Machine Learning

Background:

• Machine learning (ML) algorithms, like random forest (RF), are increasingly used for predicting patient outcomes by identifying complex data patterns.
• External information is sometimes used to enhance variable selection accuracy and model interpretability in ML.
• The benefit of external information for RF and ML prediction models remains unclear.

Purpose of the Study:

• To investigate the utility of external information from previous variable selection studies for RF models.
• To assess if external information improves prediction quality and calibration in ML models.

Main Methods:

• A plasmode simulation study was conducted using a subsampled dataset from a large pharmacoepidemiologic study (nearly 200,000 individuals).
• The study included two binary outcomes and 1152 candidate predictor variables.
• External information from traditional statistical modeling (Lasso) and univariate selection was used to reduce the scope of candidate predictors for RF models.

Main Results:

• Reducing the number of candidate predictors based on external knowledge led to improved calibration in RF models.
• Prediction quality, measured by cross-entropy, AUROC, and Brier score, did not show improvement.
• The study highlights that while calibration may benefit, overall predictive performance did not increase with externally guided variable selection.

Conclusions:

• External information can enhance the calibration of random forest models in patient outcome prediction.
• Overall prediction quality metrics (cross-entropy, AUROC, Brier score) were not improved by using external information for variable selection.
• It is crucial to critically evaluate the methodological quality of external data sources used in developing future prediction models.