A new accuracy metric under three classes when subclasses are involved and its confidence interval estimation.

Area of Science:

• Biostatistics
• Machine Learning
• Biomarker Discovery

Background:

• Compound multi-class classification involves multiple main classes and subclasses.
• Evaluating biomarker performance in this setting often uses subclasses pooling, which has limitations.

Purpose of the Study:

• To explore the downsides of accuracy metrics based on pooled data in compound multi-class classification.
• To propose a novel accuracy measure, volume under compound surface (VUS), for compound multi-class classification with three ordinal main classes.

Main Methods:

• Investigated the limitations of accuracy metrics using pooled data.
• Developed and proposed the volume under compound surface (VUS) metric.
• Studied parametric and nonparametric methods for VUS confidence interval estimation.
• Conducted simulation studies to assess coverage probabilities.
• Analyzed a subset of the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset.

Main Results:

• Identified significant downsides of accuracy metrics derived from pooled subclass data.
• The proposed VUS metric effectively evaluates biomarker accuracy without requiring subclass value ordering.
• Simulation studies demonstrated the reliability of confidence interval estimation methods for VUS.

Conclusions:

• The volume under compound surface (VUS) offers a more appropriate accuracy evaluation for compound multi-class classification compared to pooled data methods.
• VUS provides a robust approach for biomarker performance assessment in complex classification scenarios.
• The proposed methods and analysis contribute to improved statistical practices in biomarker research.