When optimism hurts: inflated predictions in psychiatric neuroimaging

Area of Science:

• Neuroimaging
• Machine Learning
• Clinical Prediction

Background:

• Neuroimaging data holds potential for predicting clinical outcomes in conditions like depression, addiction, and dementia.
• Standard neuroimaging analyses may lack the rigorous methods needed to accurately assess predictive model performance.
• Regression models are prone to overfitting sample-specific characteristics, leading to overestimated predictive ability.

Purpose of the Study:

• To highlight the critical need for robust methods to assess predictive model performance in neuroimaging.
• To demonstrate the issue of optimism in prediction models, particularly in psychiatric neuroimaging.
• To provide recommendations for the accurate assessment of model performance.

Main Methods:

• Simulated data analysis to illustrate model performance with random data.
• Review of existing literature for examples of optimistic prediction.
• Development of recommendations for evaluating predictive models.

Main Results:

• Simulations showed that models can appear predictive even with random data and outcomes.
• Examples of optimistic prediction were identified in the scientific literature.
• Overfitting is a significant concern, especially when the number of predictors is high relative to the sample size.

Conclusions:

• Accurate assessment of predictive model performance is essential for reliable neuroimaging-based predictions.
• Failure to account for model optimism can lead to misleading conclusions in clinical neuroimaging research.
• Adopting recommended assessment techniques is crucial for trustworthy clinical outcome prediction from neuroimaging data.