SERE: Exploring Feature Self-Relation for Self-Supervised Transformer.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Self-supervised learning is effective for Convolutional Neural Networks (CNNs) in vision tasks.
• Vision Transformers (ViTs) offer strong representation capabilities via spatial self-attention and feedforward networks.
• Existing self-supervised methods for ViTs often adapt CNN strategies, overlooking ViT's unique properties.

Purpose of the Study:

• To develop a novel self-supervised learning approach tailored for Vision Transformers (ViTs).
• To leverage the inherent relational modeling capabilities of ViTs across spatial and channel dimensions.
• To improve the representation power of ViTs for downstream vision tasks.

Main Methods:

• Introduced feature SElf-RElation (SERE), a novel self-supervised learning strategy for ViTs.
• Utilized spatial and channel self-relations within features for training, moving beyond instance-level discrimination.
• Focused on enhancing the relational modeling properties specific to ViT architectures.

Main Results:

• SERE significantly improved the representation learning capabilities of ViTs.
• The proposed method demonstrated stable performance enhancements across multiple downstream vision tasks.
• Self-relation based learning effectively capitalized on ViT's architectural strengths.

Conclusions:

• Feature SElf-RElation (SERE) offers a ViT-specific self-supervised learning approach.
• This method enhances ViT's relational modeling, leading to superior feature representations.
• SERE provides a promising direction for advancing self-supervised learning in Vision Transformers.