A Survey on Information Bottleneck.

Area of Science:

• Information Theory
• Machine Learning
• Pattern Analysis

Background:

• Information Bottleneck (IB) theory, developed in 1999, offers a framework for balancing data compression and information preservation.
• IB has become a popular approach for pattern analysis and representation learning.
• This survey commemorates Prof. Naftali Tishby, a key figure in IB theory.

Purpose of the Study:

• To provide a comprehensive overview of theoretical advancements and practical applications of Information Bottleneck over two decades.
• To systematically explore the fundamental theory, optimization techniques, models, and algorithms associated with IB.
• To categorize and discuss existing IB methods, including traditional and deep learning-based approaches.

Main Methods:

• Categorization of traditional IB methods into Basic, Informative, and Propagating IB based on technique.
• Classification of deep IB methods based on problem settings: understanding DNNs with IB, optimizing DNNs using IB, and DNN-based IB.
• Systematic exploration of theoretical progress and practical applications.

Main Results:

• The survey categorizes IB methods into traditional (non-neural network) and deep (neural network-involved) approaches.
• Traditional IB methods are classified by technique, while deep IB methods are classified by problem setting.
• Key theoretical and applied aspects of IB over the past 20+ years are systematically reviewed.

Conclusions:

• The survey presents a holistic view of Information Bottleneck theory and its applications.
• It highlights the evolution from traditional IB to deep IB methods integrated with deep neural networks.
• Identifies areas for future research to further advance the field of Information Bottleneck.