Stochastic Thermodynamics of Learning Parametric Probabilistic Models

Area of Science:

• Thermodynamics of Information
• Machine Learning Theory
• Information Theory

Background:

• Machine learning problems can be viewed as evolving parametric probabilistic models (PPMs).
• Thermodynamics of information offers tools to analyze information-theoretic content during learning.

Purpose of the Study:

• To assess the information-theoretic content of learning PPMs.
• To introduce novel metrics for information flow during the machine learning process.

Main Methods:

• Formulating machine learning problems as time evolution of PPMs.
• Introducing memorized information (M-info) and learned information (L-info) metrics.
• Analyzing the relationship between L-info, entropy production, and model parameters.

Main Results:

• Two information-theoretic metrics, M-info and L-info, were defined to track information flow in PPM learning.
• Accumulation of L-info correlates with entropy production.
• Model parameters function as a heat reservoir, storing learned information as M-info.

Conclusions:

• Machine learning processes can be effectively modeled using thermodynamic principles.
• The proposed metrics provide a framework for quantifying information in machine learning.
• Understanding information flow through thermodynamic analogies deepens insights into model learning and parameter behavior.