Stable reliability diagrams for probabilistic classifiers

Area of Science:

• Statistics
• Machine Learning
• Data Science

Background:

• Reliability diagrams assess forecast calibration by comparing predicted probabilities with observed frequencies.
• Classical binning methods for reliability diagrams lack statistical consistency and reproducibility due to ad hoc decisions.

Purpose of the Study:

• To introduce a novel, automated approach for generating statistically consistent and reproducible reliability diagrams.
• To provide improved tools for the diagnostics and inference of probabilistic forecasts.

Main Methods:

• The CORP approach utilizes nonparametric isotonic regression implemented via the pool-adjacent-violators (PAV) algorithm.
• Reliability diagrams are generated by plotting PAV-(re)calibrated forecast probabilities.

Main Results:

• The CORP approach produces provably statistically consistent and optimally binned reliability diagrams.
• It allows for uncertainty quantification and provides a numerical measure of miscalibration.
• A CORP-based Brier-score decomposition is introduced, generalizing to any proper scoring rule.

Conclusions:

• The CORP approach offers a robust and reproducible method for evaluating probabilistic forecast reliability.
• This method enhances diagnostic capabilities for a wide range of statistical and machine learning models.
• The PAV algorithm provides a powerful tool for improving forecast calibration and inference.