Deconstructing Cross-Entropy for Probabilistic Binary Classifiers.

Area of Science:

• Machine Learning
• Information Theory
• Decision Theory

Background:

• Cross-entropy is a fundamental metric in machine learning for evaluating classifiers.
• Its theoretical underpinnings, particularly within Bayesian decision theory and information theory, warrant deeper investigation.

Purpose of the Study:

• To provide a comprehensive analysis of the cross-entropy function from information-theoretical and Bayesian decision theory perspectives.
• To introduce a novel decomposition of cross-entropy and a visualization tool for classifier evaluation.

Main Methods:

• Contextualization of cross-entropy within Bayesian decision theory.
• Information-theoretical analysis of cross-entropy components, including prior knowledge and feature value (likelihood ratio).
• Development and application of the Empirical Cross-Entropy (ECE) plot for performance analysis.

Main Results:

• Explicit analysis of prior knowledge and feature value contributions to cross-entropy.
• Introduction of a discrimination-calibration decomposition for precise classifier performance measurement.
• Demonstration of ECE plots' efficacy in speaker verification and forensic glass analysis.

Conclusions:

• The study offers new insights into cross-entropy's meaning and interpretation.
• The discrimination-calibration decomposition enhances classifier evaluation and probability calibration strategies.
• ECE plots provide a powerful tool for visualizing and analyzing classifier performance.