Improved Estimation of Correlation Accuracy for Machine Learning Brain-Phenotype Associations

Machine learning is used in neuroscience to examine brain-phenotype associations and facilitate individual prediction from high-dimensional brain imaging. For continuous phenotypes, Pearson's correlation between the observed and predicted phenotype is used to quantify model accuracy in testing data. However, recent research suggests millions of samples may be needed to reliably estimate the maximum achievable predictive accuracy (MAPA). We formally define the MAPA and show that Pearson's estimator is biased for this quantity and its confidence intervals fail to capture the target. We develop a semiparametric (double machine learning) one-step estimator that more accurately estimates the MAPA and yields valid confidence intervals across flexible machine learning settings. Analyzing data from the Reproducible Brain Charts dataset, we show that this estimator has smaller bias when estimating brain-phenotype associations of neuroimaging data with age and psychopathology phenotypes. We show that MAPA for psychopathology factor scores using machine learning models built on structural and functional imaging measures is not better than using demographic and nuisance covariates alone.