Selective classification under imbalance in multiclass settings: A novel metric for bias-aware risk-coverage...

Area of Science:

• Machine Learning
• Medical Informatics
• Data Science

Background:

• Selective classification enhances model reliability by abstaining on uncertain inputs, crucial for safety-critical fields like healthcare.
• Existing evaluation metrics for selective classification can obscure fairness issues, particularly with imbalanced data, leading to biased rejection rates for minority classes.

Purpose of the Study:

• To introduce imbalance-aware evaluation metrics and a class-conditional selection strategy for selective classification.
• To address fairness concerns and improve the reliability of selective classifiers in imbalanced datasets, especially in clinical settings.

Main Methods:

• Proposed two imbalance-aware metrics: Class-Averaged AURC (CA-AURC) and Class-Averaged AUGRC (CA-AUGRC), integrating risk and class-specific coverage.
• Introduced Area under the IAM-coverage curve (AUIC) as a complementary metric.
• Developed a class-conditional coverage-matching selection strategy for balanced rejection across diagnostic categories.
• Evaluated the framework on a clinical dataset and three public benchmarks.

Main Results:

• Class-averaged metrics revealed significant disparities in rejection behavior under class imbalance, unlike conventional metrics.
• The proposed class-conditional strategy consistently outperformed classical approaches and a set-valued baseline across all datasets.
• Statistical comparisons confirmed significant advantages of the class-conditional strategy on CA-AUGRC and AUIC.

Conclusions:

• Class-aware evaluation and selection are essential for fairness and clinical utility in selective classification.
• Class-averaged metrics and class-conditional selection offer a more reliable assessment of selective classifiers for imbalanced medical data.
• The proposed methods demonstrate generalizability across diverse datasets and uncertainty measures.