Generalizing Information to the Evolution of Rational Belief

Area of Science:

• Information Theory
• Probability Theory
• Bayesian Inference
• Machine Learning

Background:

• Information theory quantifies uncertainty using probability distributions and measures like Shannon entropy.
• Existing measures (Kullback-Leibler divergence, cross-entropy, mutual information) are based on Shannon's concept.
• Belief is mathematically represented by probability distributions reflecting outcome plausibility.

Purpose of the Study:

• To derive a general theory of information from first principles.
• To account for evolving beliefs and unify existing information measures.
• To explore novel information measures compatible with Bayesian paradigms and machine learning.

Main Methods:

• Developed a general theory of information based on the concept of belief change.
• Reconciled existing information measures within this new theoretical framework.
• Analyzed and experimentally validated novel information measures with well-defined properties.

Main Results:

• Information is redefined as a measure of change in belief, not just uncertainty.
• Entropy is interpreted as expected information gain from a random variable realization.
• The theory accommodates evolving beliefs and is compatible with Bayesian updating.

Conclusions:

• The general theory of information provides a unified framework for understanding information measures.
• This approach offers new insights into machine learning, including quantifying model information and residual data information.
• Novel Bayesian approaches for feature selection and anomaly detection are facilitated.