A Measure of Information Available for Inference

Area of Science:

• Computational neuroscience
• Information theory
• Machine learning

Background:

• Mutual information quantifies information shared between a neural network and the external world.
• Sensory input surprise measures unpredictability and serves as an indicator of inference ability.
• The free-energy principle (FEP) posits that neural networks minimize variational free energy to perceive their environment.

Purpose of the Study:

• To investigate the role of free energy in neural networks.
• To establish free energy as a measure of the gap between stored information and information utilized for inference.
• To integrate the free-energy principle and the infomax principle for a comprehensive understanding of information processing.

Main Methods:

• Information-theoretic analysis of neural network states and external world states.
• Formulation of variational free energy as an upper bound for sensory surprise.
• Conceptual integration of the free-energy principle and the infomax principle.

Main Results:

• Free energy is demonstrated to represent the difference between the total information stored in a neural network and the information actively available for inference.
• This finding provides a novel interpretation of free energy within the context of the free-energy principle.
• The study establishes a quantitative link between information storage and inference capabilities.

Conclusions:

• Free energy serves as a critical metric for understanding how neural networks utilize stored information for real-world inference.
• Minimizing free energy is essential for efficient perception and survival, highlighting the importance of inference over mere information storage.
• The proposed measure offers a valuable tool for quantifying information available for inference in artificial and biological systems.