Enhancing Representations Through Heterogeneous Self-Supervised Learning.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Hybrid networks combining transformers and convolutions are common in vision tasks.
• The complementarity of heterogeneous architectures is underexplored in self-supervised learning.

Purpose of the Study:

• Introduce Heterogeneous Self-Supervised Learning (HSSL) to leverage architectural diversity.
• Improve representation learning in base models without structural modifications.

Main Methods:

• Enforce a base model to learn from an auxiliary head with a heterogeneous architecture.
• Experiment with various heterogeneous model pairs to analyze representation quality.
• Develop a search strategy for optimal auxiliary head selection and methods to increase model discrepancy.

Main Results:

• Representation quality of the base model improves with greater architectural discrepancy.
• HSSL is compatible with various self-supervised learning methods.
• Achieved superior performance across image classification, semantic segmentation, instance segmentation, and object detection tasks.

Conclusions:

• HSSL effectively utilizes heterogeneous architectures for enhanced self-supervised representation learning.
• Architectural discrepancy is a key factor for improving model performance.
• The proposed methods offer a flexible and effective approach for self-supervised learning in computer vision.