Effective and efficient self-supervised masked model based on mixed feature training

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Masked Image Modeling (MIM) uses attention mechanisms for masked image training, similar to Masked Language Modeling (MLM).
• Current MIM methods face high computational complexity and time costs due to extensive iterations for reconstructing masked image regions.
• Existing approaches require substantial computational resources for single-image processing in masked training paradigms.

Purpose of the Study:

• To develop an Effective and Efficient self-supervised Masked model based on Mixed feature training (EESMM).
• To reduce the computational complexity and time costs associated with masked image modeling.
• To accelerate the pre-training process for image models without sacrificing performance.

Main Methods:

• Inputting fused features from stacked images into the network to reduce computational load and enhance feature learning.
• Employing a decoding strategy that utilizes features from original and mixed images to improve representation.
• Constructing a specialized loss function during decoding to enhance feature representation quality.

Main Results:

• EESMM achieved 83% accuracy on ImageNet.
• The model completed pre-training in just 363 hours using four V100 GPUs.
• This represents a tenfold reduction in training time compared to SimMIM, demonstrating significant acceleration.

Conclusions:

• EESMM substantially accelerates the pre-training process for masked image modeling.
• The proposed method achieves competitive accuracy while drastically reducing computational time.
• EESMM offers an efficient alternative for self-supervised learning in computer vision tasks.