Machine learning prediction of cognition from functional connectivity: Are feature weights reliable?
1Melbourne Neuropsychiatry Centre, Department of Psychiatry, The University of Melbourne, Australia.
Neuroimage
|October 21, 2021
Summary
Machine learning models predict cognitive performance from brain connectivity but offer limited neurobiological insight. Feature weight reliability is poor, hindering understanding of brain processes supporting cognition.
Area of Science:
- Neuroscience
- Cognitive Science
- Machine Learning
Background:
- Machine learning accurately predicts cognitive performance from functional brain connectivity.
- Current predictive models offer limited insight into the neurobiological underpinnings of cognition.
- Reliable feature selection and weight estimation are crucial for identifying key brain circuits.
Purpose of the Study:
- Investigate the test-retest reliability of feature weights in machine learning models predicting cognitive performance.
- Assess the reliability of models built from resting-state functional connectivity networks in healthy adults.
- Determine factors influencing feature weight reliability and its relationship with prediction accuracy.
Main Methods:
- Used resting-state functional connectivity data from 400 healthy young adults.
- Built various predictive models for cognitive performance.
- Evaluated feature weight reliability using intraclass correlation coefficients (ICC).
- Investigated the impact of sample size, transformations, feature selection methods, and feature space size on reliability.
Main Results:
- Prediction accuracies were modest (r=0.2-0.4).
- Feature weight reliability was generally poor (ICC < 0.3) across models, significantly lower than for predicting biological attributes like sex (ICC ≈ 0.5).
- Larger sample sizes, Haufe transformation, non-sparse methods, and smaller feature spaces marginally improved reliability (ICC < 0.4).
- A trade-off exists between feature weight reliability and prediction accuracy; univariate statistics showed slightly better reliability than model-derived weights.
- Cross-validation fold agreement inflated reliability estimates, recommending out-of-sample reliability assessment.
Conclusions:
- Current machine learning models for cognitive prediction from brain connectivity lack reliable feature weights.
- Poor feature weight reliability limits mechanistic understanding of cognitive neurobiology.
- Shifting focus from prediction accuracy to model reliability is essential for advancing cognitive neuroscience research using machine learning.
