SCRIB: Set-Classifier with Class-Specific Risk Bounds for Blackbox Models.

Area of Science:

• Machine Learning
• Medical Informatics
• Computer Vision

Background:

• Deep learning (DL) classifiers struggle to determine when to abstain from prediction.
• Existing classification with rejection options methods do not account for varying class importance.
• Controlling prediction risk is crucial, especially in high-stakes domains like healthcare.

Purpose of the Study:

• To introduce a novel method, Set-classifier with Class-specific RIsk Bounds (SCRIB), for deep learning classification.
• To address the limitation of overlooking class-specific significance in rejection options.
• To enable DL models to make more informed decisions by assigning multiple labels and controlling class-specific risks.

Main Methods:

• SCRIB constructs a set-classifier using the output of a black-box model on a validation set.
• It assigns multiple labels to each example, enabling a rejection mechanism.
• Rejection occurs when the set classifier outputs more than one label, controlling class-specific prediction risks.

Main Results:

• SCRIB demonstrated effective control over class-specific prediction risks across medical applications.
• The method was validated on sleep staging (EEG), COVID-19 X-ray classification, and atrial fibrillation detection (ECG).
• SCRIB achieved class-specific risks 35%-88% closer to target risks compared to baseline methods.

Conclusions:

• SCRIB offers a robust solution for incorporating class-specific risk bounds into deep learning classification.
• The approach enhances the reliability of DL models in medical applications by allowing informed abstention.
• SCRIB represents a significant advancement in classification with rejection options, prioritizing critical class predictions.