Selective Classification Under Distribution Shifts.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• Selective Classification (SC) enables AI classifiers to abstain from uncertain predictions, crucial for high-stakes applications.
• Existing SC methods often assume ideal data distributions, failing to address real-world deployment challenges like distribution shifts.
• Imperfect classifiers, due to noise or robustness issues, necessitate advanced SC techniques for reliable deployment.

Purpose of the Study:

• To propose the first Selective Classification framework, termed generalized selective classification (GSC), that explicitly addresses data distribution shifts.
• To develop novel, non-training-based confidence-score functions for GSC tailored for deep learning (DL) classifiers.
• To enhance the reliability and effectiveness of SC in practical, out-of-distribution scenarios.

Main Methods:

• Developed a generalized selective classification (GSC) framework to handle in-distribution, label-shifted, and covariate-shifted samples.
• Proposed two novel margin-based confidence-score functions specifically for GSC with deep learning models.
• Focused on non-training-based score functions to avoid retraining complexities.

Main Results:

• The proposed score functions demonstrated superior effectiveness and reliability compared to existing methods for generalized SC.
• Empirical validation across various classification tasks and deep learning architectures confirmed the framework's performance.
• The study provides a robust solution for deploying classifiers under distribution shifts.

Conclusions:

• Generalized selective classification is a critical advancement for deploying AI in real-world, non-ideal conditions.
• The novel margin-based score functions offer a reliable approach for GSC in deep learning.
• This work bridges the gap between theoretical SC research and practical deployment challenges.