Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Uncertainty quantification (UQ) in deep learning is vital for high-stakes applications.
• Current UQ evaluation relies on point-prediction metrics like negative log-likelihood (NLL) and expected calibration error (ECE).
• Marginal coverage, a statistical concept, offers an intuitive alternative for assessing UQ quality but lacks systematic study in deep learning.

Purpose of the Study:

• To conduct the first large-scale evaluation of empirical frequentist coverage properties for popular UQ techniques in deep learning.
• To assess the performance of UQ methods on both regression and classification tasks.
• To investigate the impact of dataset shift on the reliability of UQ.

Main Methods:

• Empirical evaluation of well-established UQ techniques.
• Assessment across a diverse suite of regression and classification benchmark datasets.
• Analysis of marginal coverage and prediction interval width for in-distribution and out-of-distribution samples.

Main Results:

• Several UQ methods demonstrate adequate coverage for in-distribution data.
• Coverage properties significantly degrade when encountering out-of-distribution samples.
• The performance of UQ techniques diminishes as dataset shift increases.

Conclusions:

• Current UQ methods in deep learning are not robust to dataset shift.
• Marginal coverage is a critical metric for evaluating UQ reliability in real-world scenarios.
• Further research is needed to develop UQ techniques that maintain performance under data distribution changes.