Classification based hypothesis testing in neuroscience: Below-chance level classification rates and overlooked...

Area of Science:

• Neuroscience
• Machine Learning
• Statistical Analysis

Background:

• Multivariate pattern analysis (MVPA) is widely used for analyzing complex datasets, particularly in neuroscience.
• Correct classification rate (CCR) is a common metric to quantify classification accuracy and effect size in MVPA.
• Standard analysis assumes a symmetric distribution of CCR, which may not hold true for certain data characteristics.

Purpose of the Study:

• To investigate the distribution of CCR in multivariate pattern analysis (MVPA) for low sample size (LSS) and low effect size (LES) data, common in neuroscience.
• To evaluate the suitability of CCR for hypothesis testing and effect size estimation in these challenging data conditions.
• To propose alternative methods for reporting MVPA results that are more robust to data limitations.

Main Methods:

• Simulated low sample size, low effect size data typical of neuroscience studies.
• Cross-validation of linear multivariate pattern analysis (MVPA).
• Analysis of the distribution of correct classification rates (CCRs) and comparison with chance levels.
• Estimation of P values using randomization tests.

Main Results:

• The distribution of CCRs in LSS/LES data is asymmetric, with a mode above chance but rates considerably below chance.
• CCR does not accurately reflect the true effect size in these conditions.
• The skewness of the null distribution of CCRs invalidates many standard parametric tests.
• Cross-validation with fewer folds (e.g., twofold) is more sensitive despite lower average CCRs.

Conclusions:

• Correct classification rate (CCR) is an unreliable metric for hypothesis testing and effect size estimation in low sample size, low effect size neuroscience data.
• Standard parametric tests are inappropriate for assessing the significance of CCRs due to skewed null distributions.
• P values derived from randomization tests provide a more robust measure for MVPA hypothesis testing.
• Cross-validation fold number impacts sensitivity, with fewer folds potentially offering better detection of effects.