Contrastive Learning via Variational Information Bottleneck

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Self-supervised learning (SSL) has advanced significantly, particularly with contrastive learning methods.
• Contrastive learning aims to maximize mutual information between image augmentations (positive pairs).
• However, this objective can lead to over-confidence and capture spurious correlations, degrading representation quality.

Purpose of the Study:

• To address the limitations of vanilla contrastive learning by reducing superfluous relevance between positive views.
• To introduce a novel regularization technique to improve the quality of learned representations in SSL.

Main Methods:

• Introduced representation entropy minimization regularization to vanilla contrastive learning.
• Derived an analytical expression for the objective by framing it as an information bottleneck problem.
• Solved the objective via variational approximation, resulting in the CLIMB framework.

Main Results:

• CLIMB (Contrastive Learning via variational InforMation Bottleneck) consistently improves performance across benchmarks.
• When instantiated with DINO, CLIMB achieved notable gains: 4.5% and 3.5% on k-NN classification with EfficientNet-B0 and ResNet-50 backbones, respectively.

Conclusions:

• The proposed CLIMB framework effectively alleviates superfluous relevance in contrastive learning.
• Minimizing representation entropy leads to more robust and informative feature representations in self-supervised models.