Widespread use of invalid statistical tests in biomedical machine learning.

Area of Science:

• Biomedical research
• Machine learning
• Statistical inference

Background:

• Machine learning accelerates biomedical research, with cross-validation commonly used for performance comparison.
• Standard statistical tests for comparing prediction performance assume independence, which is violated by cross-validation folds.
• Ignoring this fold dependence inflates false positive rates, compromising research validity.

Purpose of the Study:

• To investigate the prevalence and impact of ignoring cross-validation fold dependence in biomedical machine learning.
• To propose a novel statistical method for valid model comparison that accounts for fold dependence.

Main Methods:

• A PRISMA-guided meta-analysis of 210 studies was conducted to assess reporting practices.
• Simulations across 420 scenarios with diverse datasets evaluated the impact of fold dependence and tested new methods.
• The proposed SHARP (Split-HAlf RePeated) test was developed as a modification to standard cross-validation.

Main Results:

• 97% of studies ignored fold dependence when comparing prediction performance, a problem pervasive across fields.
• Ignoring fold dependence leads to invalid false positive control, especially with repeated cross-validation.
• The SHARP test demonstrated superior performance in controlling false positives, maintaining statistical power, and calibrating confidence intervals compared to 12 other tests.

Conclusions:

• The widespread failure to account for cross-validation fold dependence undermines the reliability of machine learning research in biomedicine.
• The SHARP test offers a robust solution for valid statistical inference in model comparison.
• Best practices and reporting guidelines are needed to ensure accurate model comparison in biomedical machine learning.