Related Experiment Video
Updated: Jul 10, 2026
14:27
Identification of Disease-related Spatial Covariance Patterns using Neuroimaging Data
Published on: June 26, 2013
15.8K
Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics
Andrew Cwiek1,2, Sarah M Rajtmajer3,4, Bradley Wyble1
1Department of Psychology, Pennsylvania State University, University Park, PA, USA.
Network Neuroscience (Cambridge, Mass.)
|March 30, 2022
Summary
Machine learning (ML) models applied to resting-state functional MRI (fMRI) show reduced accuracy without "lockbox" validation. Improving transparency and using lockbox data are crucial for reliable neuroimaging biomarkers.
Area of Science:
- Neuroscience
- Artificial Intelligence
- Medical Imaging
Background:
- Machine learning (ML) is increasingly used to interpret functional neuroimaging data, particularly resting-state functional MRI (fMRI).
- Predictive models aim to infer dimensions of the human functional connectome.
Purpose of the Study:
- To critically review the application of ML in functional neuroimaging.
- To assess common practices and identify methodological pitfalls in ML model training and evaluation.
Main Methods:
- Systematic review of 250 studies utilizing ML and resting-state fMRI.
- Analysis of model performance metrics, focusing on cross-validation versus holdout ('lockbox') data.
- Assessment of transparency in model development and evaluation.
Main Results:
- Holdout ('lockbox') performance was approximately 13% less accurate than cross-validation alone.
- Only 16% of reviewed studies incorporated essential lockbox data for validation.
- A significant lack of transparency was observed in key ML model training and evaluation steps.
Conclusions:
- The use of lockbox validation is critical for accurate assessment of ML model generalizability in neuroimaging.
- Addressing methodological pitfalls, such as lack of transparency and inadequate validation, is essential for the neuroimaging community.
- Recommendations are provided for integrating ML into clinical neurosciences to advance imaging biomarkers and understand brain disorders.
