Not just "big" data: Importance of sample size, measurement error, and uninformative predictors for developing...

Area of Science:

• Digital health
• Machine learning in healthcare
• Mental health treatment

Background:

• Efficient mental health care systems are in high demand.
• Digital interventions and predictive models are key to improving treatment prognosis.
• Machine learning (ML) models show promise for predicting treatment outcomes.

Purpose of the Study:

• To review ML model applications for predicting treatment prognosis, focusing on digital interventions.
• To assess the impact of sample size and measurement error on ML model performance for treatment prediction.
• To explore the role of digital interventions in facilitating large-scale studies for treatment prediction.

Main Methods:

• Simulations were used to evaluate ML model performance under varying conditions.
• The study considered complex interactions, non-linear effects, and measurement error in predictors and outcomes.
• Performance of decision tree ensembles (e.g., gradient boosted machines) was compared to regularized linear regression.

Main Results:

• Optimizing complex prediction models requires sample sizes in the thousands.
• Advanced ML methods like gradient boosted machines excel in large samples with minimal measurement error.
• In the presence of moderate measurement error, advanced ML methods offered no significant advantage over regularized linear regression, even with N=100,000.

Conclusions:

• Digital interventions are vital for scalable data collection needed for robust treatment prediction models.
• Optimal measurement practices are essential when using digital interventions for research.
• Implementing these predictive algorithms into clinical practice requires careful consideration of data quality and sample size.