Multivariate classification of neuroimaging data with nested subclasses: Biased accuracy and implications for...

Area of Science:

• Neuroscience
• Computational Biology
• Machine Learning

Background:

• Biological data often presents high dimensionality and low effect sizes, necessitating advanced analytical methods.
• Multivariate pattern analysis (MVPA), especially pattern classification, is crucial for detecting subtle differences between experimental conditions.
• Real-world data frequently contains nested subclasses within main classes, such as multiple data points from individual subjects.

Purpose of the Study:

• To investigate the impact of nested subclasses on the accuracy of multivariate pattern classification.
• To analytically and empirically demonstrate how subclasses introduce bias in classification rates.
• To propose methods for accounting for subclass bias in statistical significance testing.

Main Methods:

• Analytical proof of subclass bias on linear classifier performance.
• Simulations to evaluate bias under varying effect sizes and subclass variances.
• Application to human EEG datasets to illustrate practical implications.

Main Results:

• Nested subclasses systematically inflate correct classification rates (CCRs) beyond expected differences.
• Subclass bias is most pronounced with low between-class effect size and high subclass variance.
• Increasing subclass numbers can mitigate bias, but explicit consideration via permutation tests is essential for accurate significance.

Conclusions:

• Standard statistical tests and naive permutation methods can misestimate significance in the presence of subclass structure.
• Accurate assessment of classification performance in biological data requires methods that account for nested subclasses.
• Proper statistical validation is critical for reliable findings in MVPA of complex biological datasets, including human EEG.