Correspondence between thermodynamics and inference.

Area of Science:

• Statistical Physics
• Bayesian Inference
• Machine Learning

Background:

• A natural analogy exists between Bayesian statistics and statistical physics, where sample size mirrors inverse temperature.
• Existing methods face challenges in defining objective priors, especially with unknown model dimensions.

Purpose of the Study:

• To define and analyze novel statistical quantities: learning capacity and Gibbs entropy.
• To leverage these quantities to gain insights into model learning mechanisms.
• To develop a generalized principle of indifference for objective Bayesian prior definition.

Main Methods:

• Establishing an analogy between sample size and inverse temperature.
• Defining learning capacity analogous to heat capacity.
• Defining Gibbs entropy for counting distinguishable distributions.
• Applying the generalized principle of indifference to Bayesian inference.

Main Results:

• The learning capacity provides insight into learning mechanisms and explains high learning performance in certain models.
• The Gibbs entropy offers a method for counting distinguishable distributions.
• A generalized principle of indifference yields an objective prior definition.
• The approach accommodates unknown model dimensions and avoids rejecting higher-dimensional models.

Conclusions:

• The Bayesian statistics-statistical physics analogy offers a powerful framework for understanding learning and defining objective priors.
• This work provides a solution to the long-standing problem of objective prior definition in Bayesian inference.
• The proposed methods are applicable to complex scenarios, including those with unknown model dimensions.