Relationship between prediction accuracy and feature importance reliability: An empirical and theoretical study

Area of Science:

• Neuroimaging
• Computational Neuroscience
• Psychometrics

Background:

• Neuroimaging data is increasingly used to predict behavior.
• Feature importance quantifies predictive relevance but has shown low reliability.
• A potential trade-off exists between prediction accuracy and feature importance reliability.

Purpose of the Study:

• To investigate the universality of the trade-off between prediction accuracy and feature importance reliability.
• To determine if sufficient sample size can improve feature importance reliability.
• To compare the reliability of different feature importance methods.

Main Methods:

• Utilized a large sample size (2600 participants).
• Employed Haufe-transformed weights for feature importance estimation.
• Compared reliability with original regression weights and univariate FC-behavior correlations.
• Analyzed split-half reliability using intra-class correlation coefficients.

Main Results:

• Haufe-transformed weights achieved fair to excellent split-half reliability (ICC: 0.75 for cognitive, 0.57 for personality, 0.53 for mental health).
• Reliability was significantly higher than original regression weights and univariate correlations.
• Feature importance reliability positively correlated with prediction accuracy across phenotypes.
• Mathematical analysis showed reliability is necessary for low feature importance error.

Conclusions:

• Sufficient sample size enhances feature importance reliability in neuroimaging.
• Haufe-transformed weights offer a reliable method for interpreting predictive models.
• Improved feature importance reliability may lead to better prediction accuracy.
• Findings provide empirical and theoretical insights into neuroimaging-based behavioral prediction.