Teaching Masked Autoencoder With Strong Augmentations.

Area of Science:

• Computer Vision
• Machine Learning
• Self-Supervised Learning

Background:

• Masked Autoencoders (MAE) are effective self-supervised learners but struggle with high-level discriminability, leading to poor linear probing performance.
• Strong data augmentation, crucial in contrastive learning, presents challenges for MAE due to pixel uncertainty affecting reconstruction.
• Existing methods often see performance degradation when directly applying strong augmentation to MAE.

Purpose of the Study:

• To investigate the potential of strong augmented views to enhance MAE's discriminability while preserving its reconstruction advantages.
• To propose a novel approach that integrates strong augmentation into MAE without compromising its core functionalities.

Main Methods:

• Introduced Masked Siamese Autoencoder (MSA), a model featuring a student and a teacher branch.
• The student branch utilizes MAE's architecture, while the teacher branch uses an unmasked strong view as a teacher signal.
• The teacher branch imposes high-level discrimination onto the student branch, guiding its learning process.

Main Results:

• MSA improves the model's spatial perception and global inter-image discrimination capabilities.
• Pretraining with MSA yields superior performance across various downstream tasks compared to standard MAE.
• Achieved 6.1% gain in linear probing on ImageNet-1k and 67.4% accuracy on VQAv2, outperforming DeiT and MAE.

Conclusions:

• The proposed MSA effectively leverages strong augmentation to enhance MAE's discriminative power.
• MSA offers a simple yet effective solution for improving self-supervised learning models.
• The method demonstrates significant improvements in both feature representation and downstream task performance.