H-Calibration: Rethinking Classifier Recalibration With Probabilistic Error-Bounded Objective.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Probability Theory

Background:

• Deep neural networks (DNNs) excel in various tasks but frequently exhibit miscalibration, leading to unreliable probability outputs.
• Post-hoc recalibration methods aim to improve DNN probability reliability without compromising classification performance.
• Existing methods face limitations, motivating the development of more robust calibration techniques.

Purpose of the Study:

• To categorize and analyze limitations of current post-hoc calibration methods for DNNs.
• To propose a novel probabilistic learning framework, $h$-calibration, for enhanced model calibration.
• To develop an effective post-hoc calibration algorithm based on the $h$-calibration framework.

Main Methods:

• Categorization and theoretical/practical analysis of existing calibration strategies (intuitive, binning-based, ideal calibration formulations).
• Development of the $h$-calibration probabilistic learning framework, establishing an equivalent learning formulation for canonical calibration with boundedness.
• Design and implementation of a post-hoc calibration algorithm derived from the $h$-calibration framework.

Main Results:

• Identified and addressed ten common limitations in previous post-hoc calibration approaches.
• The proposed $h$-calibration algorithm demonstrated superior performance compared to traditional methods in extensive experiments.
• Achieved state-of-the-art results on standard post-hoc calibration benchmarks, validating theoretical effectiveness.

Conclusions:

• The $h$-calibration framework provides a theoretically sound and practically effective approach to learning error-bounded calibrated probabilities.
• The research elucidates the convergence properties of computational statistics concerning theoretical bounds in canonical calibration.
• This work offers a valuable reference for improving reliable likelihood estimation in machine learning and related fields.