Data leakage inflates prediction performance in connectome-based machine learning models.

Area of Science:

• Neuroimaging
• Machine Learning
• Computational Neuroscience

Background:

• Predictive modeling in neuroimaging aims to uncover brain-behavior relationships and ensure generalizability.
• Data leakage, a breach in training-test data separation, compromises model validity and is prevalent in machine learning.
• Understanding leakage effects is vital for evaluating existing neuroimaging literature.

Purpose of the Study:

• To investigate the impact of five distinct data leakage types on neuroimaging predictive models.
• To assess how leakage affects functional and structural connectome-based machine learning.
• To determine the influence of dataset size on leakage effects.

Main Methods:

• Examined five forms of data leakage: feature selection, covariate correction, and inter-subject dependencies.
• Applied machine learning models to functional and structural connectome data across four datasets.
• Evaluated model performance on three distinct phenotypes.

Main Results:

• Leakage through feature selection and repeated subjects significantly inflated prediction performance.
• Other leakage forms demonstrated minimal impact on predictive model accuracy.
• Smaller datasets amplified the detrimental effects of data leakage.

Conclusions:

• Data leakage has variable effects on neuroimaging predictive models, with some forms being more detrimental than others.
• Feature selection and subject duplication are critical leakage points to avoid.
• Ensuring data integrity by preventing leakage is essential for enhancing the validity and reproducibility of neuroimaging research.