Efficient Masked Autoencoders With Self-Consistency.

Area of Science:

• Computer Vision
• Self-Supervised Learning

Background:

• Masked Image Modeling (MIM) is a powerful self-supervised pre-training technique in computer vision, inspired by Masked Language Modeling (MLM).
• High random mask ratios in MIM lead to inefficient data utilization and inconsistent predictions, hindering pre-training speed and reliability.

Purpose of the Study:

• To introduce Efficient Masked Autoencoders with Self-Consistency (EMAE) to enhance MIM pre-training efficiency and prediction consistency.
• To address the limitations of prolonged pre-training and unreliable generations caused by high mask ratios in traditional MIM.

Main Methods:

• A parallel mask strategy divides images into K parts, each masked and processed in parallel within an iteration.
• Self-consistency learning is employed to ensure consistent predictions for overlapping masked patches across different parts.
• The model minimizes the loss between predictions and masked patches during parallel processing.

Main Results:

• EMAE significantly improves pre-training efficiency, achieving top performance on ViT-Large with only 13% of MAE pre-training time on ImageNet.
• The method demonstrates superior data utilization and generates more reliable image representations.
• EMAE achieves state-of-the-art transferability across diverse downstream tasks, including image classification, object detection, and semantic segmentation.

Conclusions:

• EMAE offers a more efficient and consistent approach to masked image modeling pre-training.
• The proposed parallel mask strategy and self-consistency learning effectively overcome the limitations of traditional MIM.
• EMAE provides a robust foundation for high-performance computer vision models across various applications.