The Deterministic Information Bottleneck

Area of Science:

• Machine Learning
• Information Theory
• Data Compression

Background:

• Lossy compression and clustering rely on identifying relevant features.
• The Information Bottleneck (IB) method formalizes this using mutual information for optimal trade-offs.
• Existing IB methods often use stochastic encoders (soft clustering).

Purpose of the Study:

• Introduce a novel formulation, the Deterministic Information Bottleneck (DIB), using entropy instead of mutual information.
• Compare DIB's performance and efficiency against the traditional IB method.
• Develop a method for continuous interpolation between soft and hard clustering.

Main Methods:

• Formulated the Deterministic Information Bottleneck (DIB) by replacing mutual information with entropy.
• Conducted comparative experiments on synthetic datasets.
• Analyzed computational efficiency across various convergence parameters.

Main Results:

• DIB and IB showed similar performance on the IB cost function.
• DIB significantly outperformed IB on the DIB cost function.
• DIB demonstrated considerable computational efficiency gains over IB.

Conclusions:

• The DIB provides a more effective compression measure using entropy.
• DIB's deterministic nature leads to hard clustering, offering computational advantages.
• The DIB framework facilitates a spectrum of clustering approaches from soft to hard.