Prediction and outlier detection in classification problems.

Area of Science:

• Machine Learning
• Statistical Learning Theory
• Data Science

Background:

• Standard multi-class classification assumes training and test data follow identical distributions, which is often violated in real-world scenarios.
• Outlier detection and robust prediction set construction are crucial for reliable decision-making when data distributions shift.

Purpose of the Study:

• To propose a novel method, Balanced and Conformal Optimized Prediction Sets (BCOPS), for multi-class classification under distribution shifts.
• To optimize prediction sets for out-of-sample performance, balancing correct class inclusion and outlier detection.
• To provide finite sample coverage guarantees without requiring distributional assumptions.

Main Methods:

• BCOPS combines supervised learning algorithms with conformal prediction principles.
• It constructs prediction sets C(x) that are subsets of class labels, potentially empty to indicate outliers.
• The method minimizes a misclassification loss averaged over the out-of-sample distribution.

Main Results:

• BCOPS provides a finite sample coverage guarantee for prediction sets, irrespective of distributional assumptions.
• The method demonstrates the ability to detect outliers by returning an empty prediction set.
• Asymptotic consistency and optimality of the proposed methods are proven under stated assumptions.

Conclusions:

• BCOPS offers a robust framework for multi-class classification when training and test data distributions differ.
• The method enhances prediction reliability by incorporating outlier detection and providing coverage guarantees.
• The proposed outlier detection rate estimation method aids in evaluating classification procedure performance.