Establishment of Best Practices for Evidence for Prediction: A Review

Area of Science:

• Neuroscience
• Psychiatry
• Biostatistics

Background:

• High-dimensional data, including neuroimaging and genomics, holds promise for predicting neuropsychiatric disease states and treatment outcomes.
• Existing studies often conflate correlation with prediction, leading to overstated claims of predictive accuracy.

Purpose of the Study:

• To identify and discuss potential problems in predictive modeling for neuropsychiatric research.
• To clarify the distinction between correlation and statistical prediction.
• To recommend best practices for robust predictive modeling.

Main Methods:

• Review of statistical methodologies for predictive modeling in high-dimensional data.
• Discussion of appropriate performance measures for classification and regression.
• Emphasis on the necessity of independent data for validation.

Main Results:

• Correlation is often inappropriately used as a measure of prediction.
• Commonly used performance measures may have limitations; multiple measures are essential.
• Area under the receiver operating characteristic curve is suitable for classification; median absolute error is preferred over correlation for regression.

Conclusions:

• Rigorous validation using separate datasets is crucial for establishing predictive validity.
• Best practices include avoiding in-sample fit indices, encompassing all operations in cross-validation, using sufficiently large samples (hundreds of observations), reporting multiple accuracy measures, using the sums of squares formulation for the coefficient of determination, and employing k-fold cross-validation.