MVEB: Self-Supervised Learning With Multi-View Entropy Bottleneck.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Self-supervised learning (SSL) aims to create generalizable representations from unlabeled data.
• Current SSL methods often assume shared information between image views is sufficient for downstream tasks.
• Discarding non-shared information can enhance representation generalization.

Purpose of the Study:

• To develop an effective method for learning minimal sufficient representations in SSL.
• To address the intractability of maximizing mutual information for representation learning.
• To propose a novel objective for improved generalization.

Main Methods:

• Introduced the Multi-View Entropy Bottleneck (MVEB) objective.
• Simplified minimal sufficient representation learning by maximizing view embedding agreement.
• Incorporated maximization of embedding distribution's differential entropy.

Main Results:

• MVEB significantly enhances performance on downstream tasks.
• Achieved 76.9% top-1 accuracy on ImageNet using a ResNet-50 backbone.
• Established a new state-of-the-art result for ResNet-50 on ImageNet linear evaluation.

Conclusions:

• MVEB provides an effective approach to learning minimal sufficient representations.
• The proposed method improves generalization capabilities of self-supervised models.
• MVEB sets a new benchmark for representation learning with ResNet-50.