MV-MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Self-supervised learning (SSL) is crucial for leveraging unlabeled data in machine learning.
• Knowledge distillation aims to transfer knowledge from a larger model to a smaller one.
• Existing SSL methods often rely on contrastive learning, clustering, or stop gradients, which can be limiting.

Purpose of the Study:

• To introduce a new self-supervised learning and knowledge distillation framework named multi-views and multi-representations (MV-MR).
• To demonstrate the efficacy of MV-MR for efficient self-supervised classification and model-agnostic knowledge distillation.
• To showcase MV-MR's ability to incorporate constraints on learnable embeddings using image multi-representations as regularizers.

Main Methods:

• The MV-MR method maximizes dependence between learnable embeddings from augmented and non-augmented views.
• It also maximizes dependence between learnable embeddings from augmented views and non-learnable representations from non-augmented views.
• The framework avoids contrastive learning, clustering, and stop gradients, offering a generic approach.

Main Results:

• MV-MR achieves state-of-the-art self-supervised performance on STL10 and CIFAR20 datasets in a linear evaluation setup.
• A ResNet50 model, pretrained using MV-MR knowledge distillation with a CLIP ViT model, attains state-of-the-art results on STL10 and CIFAR100.
• The method proves effective for efficient self-supervised classification and model-agnostic knowledge distillation.

Conclusions:

• The MV-MR framework offers a novel and effective approach to self-supervised learning and knowledge distillation.
• It achieves superior performance compared to existing methods, particularly in linear evaluation settings.
• MV-MR provides a flexible and powerful tool for representation learning and model compression.