Key Measures for Evaluating Diagnostic Accuracy in Multi-Class Classification: An Overview and Simulation-Based...

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Statistical Modeling

Background:

• AI advancements drive predictive modeling in medicine.
• Need for robust multi-class classification metrics due to system complexity.
• Limited comparative studies on multi-class metrics under varied data conditions.

Purpose of the Study:

• To provide an overview of common multi-class classification accuracy metrics.
• To systematically evaluate diagnostic accuracy measures via simulation.
• To offer practical guidance for metric selection in multi-class tasks.

Main Methods:

• Overview of established multi-class classification metrics.
• Simulation study across diverse scenarios (3- and 5-class, balanced/imbalanced data, varying predictor distributions).
• Assessment of bias and 95% confidence interval coverage for each metric.

Main Results:

• Most metrics showed stable, unbiased performance under balanced conditions.
• Imbalanced conditions revealed greater bias; M-index and polytomous discrimination index performed more stably.
• Micro-averaged ROC curve area consistently exhibited higher bias with class imbalance.

Conclusions:

• Metric performance varies significantly with data balance.
• M-index and polytomous discrimination index are recommended for imbalanced multi-class medical data.
• Systematic evaluation aids informed metric selection in AI-driven medical diagnostics.