Statistical variability in comparing accuracy of neuroimaging based classification models via cross validation.

Area of Science:

• Biomedical research
• Machine learning applications
• Neuroimaging analysis

Background:

• Machine learning (ML) advances classification accuracy in biomedical research.
• Rigorous comparison of ML model accuracy is essential but challenging.
• Cross-validation (CV) introduces complexities in statistical significance testing for ML models.

Purpose of the Study:

• To highlight practical challenges in quantifying statistical significance of accuracy differences between ML models using CV.
• To propose an unbiased framework for assessing the impact of CV setups on statistical significance.
• To address the need for more rigorous practices in biomedical ML research to mitigate reproducibility issues.

Main Methods:

• Developed an unbiased framework to evaluate the influence of CV configurations on statistical significance.
• Applied the framework to three public neuroimaging datasets.
• Analyzed the impact of data properties, testing procedures, and CV choices on detecting significant differences.

Main Results:

• Demonstrated substantial variability in detecting significant ML model differences based on data properties, testing procedures, and CV configurations.
• Re-emphasized known flaws in current p-value computations for comparing model accuracies.
• Showed that factors influencing significance are often overlooked in ML-based biomedical studies.

Conclusions:

• Variability in significance testing can lead to p-hacking and inconsistent conclusions on model improvement.
• More rigorous practices are urgently needed for comparing ML models in biomedical research.
• Implementing robust comparison methods is crucial for addressing the reproducibility crisis in the field.