Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning

Area of Science:

• Information Theory
• Deep Learning
• Machine Learning

Background:

• A new trend in deep learning utilizes similarity functions and Estimated Mutual Information (EMI) as learning objectives.
• EMI is fundamentally the same as Semantic Mutual Information (SeMI), a concept introduced 30 years ago.

Purpose of the Study:

• To review the history of semantic information measures and learning functions.
• To introduce the author's G theory of semantic information and its applications.
• To explore the relationship between SeMI, Shannon's Mutual Information (MI), and various machine learning concepts.

Main Methods:

• Review of evolutionary histories of semantic information and learning functions.
• Introduction of the G theory with the rate-fidelity function R(G).
• Application of G theory to multi-label learning, MI classification, and mixture models.

Main Results:

• Demonstration that mixture models and Restricted Boltzmann Machines converge due to SeMI maximization and Shannon's MI minimization.
• Identification of information efficiency G/R approaching 1 in these models.
• Exploration of SeMI as a reward function for reinforcement learning.

Conclusions:

• The G theory provides an interpretive framework for deep learning.
• Combining semantic information theory and deep learning can accelerate advancements in both fields.
• Potential to simplify deep learning through Gaussian channel mixture models for pre-training.