How to compute reliability estimates and display confidence and tolerance intervals for pattern classifiers using the...

Area of Science:

• Neuroscience
• Cognitive Neuroscience
• Machine Learning

Background:

• Pattern classifiers in brain imaging analyze data as discriminability or distance between categories.
• Multidimensional scaling (MDS) visualizes category configurations but lacks reliability estimates for inference.
• Current methods do not provide reliable estimates for the positions of experimental categories in brain imaging data.

Purpose of the Study:

• To present a novel procedure for estimating the reliability of pattern classifiers used in brain imaging analysis.
• To develop a method that integrates bootstrap estimation with a 3-way extension of MDS (DISTATIS) for enhanced data interpretation.
• To provide statistically sound reliability estimates for category positions in brain data visualizations.

Main Methods:

• Utilized bootstrap estimation to quantify the variability of experimental conditions in brain imaging data.
• Introduced DISTATIS, a 3-way extension of multidimensional scaling, to integrate bootstrap-generated distance matrices.
• Developed tolerance and confidence intervals for reliability estimation, including a Bonferonni-like correction for multiple comparisons.

Main Results:

• The proposed procedure yields reliability estimates (tolerance and confidence intervals) for pattern classifier results.
• The methodology allows for inferential analysis of category configurations derived from brain imaging data.
• Demonstrated the application of the method using a pattern classifier on fMRI data of object and face recognition.

Conclusions:

• The new procedure enhances the inferential power of brain imaging data analysis by providing reliability estimates.
• DISTATIS combined with bootstrap estimation offers a robust framework for visualizing and interpreting brain data patterns.
• This approach addresses limitations in existing methods, enabling more accurate conclusions from neuroimaging studies.