Beyond the hype: A simulation study evaluating the predictive performance of machine learning models in psychology

Area of Science:

• Psychological research
• Computational methods
• Data science

Background:

• Machine learning (ML) methods are increasingly popular in psychology.
• Their utility is debated due to data limitations like small sample sizes and measurement error.
• Psychological data quality often falls short of ML requirements.

Purpose of the Study:

• To examine the data requirements for effective ML performance in psychological research.
• To compare various ML models under different data conditions.
• To determine the optimal conditions for ML predictive performance.

Main Methods:

• Simulation study comparing elastic net regressions, random forests, and gradient boosting machines.
• Varied conditions included sample size, irrelevant predictors, predictor reliability, effect size, and data-generating processes (linear vs. nonlinear).
• Assessed models' ability to reach maximum attainable predictive performance.

Main Results:

• Optimal conditions (N=1,000, perfect reliability, large effect size R²=.80) rarely met in practice.
• No single ML model was consistently superior across all tested data characteristics.
• Predictive performance was fundamentally limited by data quality.

Conclusions:

• Psychological data quality is a critical bottleneck for ML predictive performance.
• Careful consideration of data characteristics is essential when applying ML in psychology.
• Comparisons between flexible ML and simpler models require nuanced interpretation based on data quality.