Common component classification: what can we learn from machine learning?

Area of Science:

• Neuroimaging
• Machine Learning
• Computational Neuroscience

Background:

• Machine learning (ML) methods achieve high accuracy (≥90%) in classifying functional magnetic resonance imaging (fMRI) scans.
• Concerns exist regarding ML classifier robustness to task reordering, reproducibility, and the influence of artifactual noise versus true neural signals.

Purpose of the Study:

• To rigorously evaluate the true performance and limitations of ML classifiers in fMRI data analysis.
• To investigate the impact of physiological noise, task reordering, and cross-scan variability on classification stability.
• To identify and quantify sources of bias in fMRI classification models and their implications for statistical inference.

Main Methods:

• Deconstruction of ML classifiers to assess sensitivity to noise and task variations.
• Training and testing models within and across runs to evaluate stability and reproducibility.
• Utilizing independent components analysis (ICA) for feature extraction and artifact removal.
• Simulating flawed models to measure bias in cross-validation estimates of testing error.

Main Results:

• Artifact removal, even with ICA, can paradoxically decrease predictive accuracy in fMRI classification.
• Flawed feature selection processes introduce bias, leading to an overestimation of model performance (cross-validation error vs. testing error).
• Small or unbalanced sample sizes in fMRI studies can significantly affect Type 1 and Type 2 error rates.

Conclusions:

• ML classification of fMRI data is susceptible to various biases and noise sources, challenging its reliability and reproducibility.
• Careful consideration of feature selection, artifact handling, and statistical assumptions is crucial for accurate fMRI classification.
• Publication bias and small sample sizes can create a false sense of confidence in ML-based fMRI findings.