Selective prediction-set models with coverage rate guarantees.

Area of Science:

• Machine Learning in Healthcare
• Medical Informatics
• Artificial Intelligence in Medicine

Background:

• Current machine learning (ML) in healthcare requires full clinician oversight or operates without human input.
• This binary approach limits ML reliability and increases the burden on healthcare professionals.
• A middle ground is needed to balance ML predictions with human expertise.

Purpose of the Study:

• To develop a framework for selective prediction-set (SPS) models that can abstain from making predictions.
• To improve the reliability of ML algorithms in healthcare by allowing them to abstain on difficult cases.
• To reduce the workload on human experts by focusing their attention on cases requiring oversight.

Main Methods:

• Introduced a general penalized loss minimization framework for training SPS models.
• Developed a model-agnostic statistical inference procedure for evaluating the coverage rate of SPS models.
• Ensembled individual SPS models trained using K-fold cross-validation for robust performance evaluation.

Main Results:

• SPS models abstain from predictions when outcomes are difficult to predict accurately, particularly for out-of-distribution data.
• Models achieve higher predictive accuracy on cases where they do provide a prediction.
• SPS ensembles demonstrated coverage rates closer to the nominal level with narrower confidence intervals.

Conclusions:

• Selective prediction-set (SPS) models offer a promising approach to enhance ML reliability in healthcare.
• The ability to abstain improves accuracy and reduces the burden on human experts.
• This method shows potential for improving diagnostic accuracy, as demonstrated in ICU patient data and MNIST image prediction.