How to evaluate uncertainty estimates in machine learning for regression?

Area of Science:

• Machine Learning
• Artificial Intelligence
• Statistical Modeling

Background:

• Neural networks are increasingly used, necessitating reliable uncertainty estimation.
• Existing methods for evaluating uncertainty estimates (log-likelihood for densities, coverage for prediction intervals) have limitations.
• These methods struggle to disentangle components of predictive uncertainty and compare diverse estimation approaches.

Purpose of the Study:

• To identify and analyze the fundamental flaws in current methods for evaluating uncertainty estimates from neural networks.
• To demonstrate the limitations of log-likelihood and direct prediction interval testing.
• To propose a novel, simulation-based approach for more robust and comparable evaluation of uncertainty quantification.

Main Methods:

• Theoretical analysis of existing evaluation metrics (log-likelihood, prediction interval coverage).
• Simulations to demonstrate the shortcomings of current methods, including issues with marginal vs. pointwise coverage.
• Development and proposal of a simulation-based testing framework for uncertainty quantification.

Main Results:

• Both log-likelihood and direct prediction interval testing exhibit significant flaws.
• Current methods cannot reliably assess individual components of predictive uncertainty.
• Testing on a single dataset can mask undesirable behaviors like over/underconfidence.
• A better log-likelihood does not guarantee improved prediction intervals.

Conclusions:

• Existing methods for evaluating neural network uncertainty estimates are inadequate.
• A new simulation-based approach is proposed to overcome these limitations.
• This new approach facilitates better comparison and development of uncertainty quantification methods.