Exploring the Coordination of Frequency and Attention in Masked Image Modeling

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Masked Image Modeling (MIM) is a key self-supervised learning technique in computer vision.
• Current MIM methods suffer from long pre-training times due to random patch masking and large datasets.
• Random masking fails to utilize semantic information, hindering effective visual representation learning.

Purpose of the Study:

• To improve the efficiency and performance of masked image modeling.
• To develop a strategy that leverages semantic information for more effective visual representation learning.
• To reduce the computational cost associated with MIM pre-training.

Main Methods:

• Proposed Frequency & Attention-driven Masking and Throwing Strategy (FAMT).
• Utilized self-attention and frequency domain information to identify and mask semantic image patches.
• Implemented a patch throwing strategy to decrease the number of training patches.

Main Results:

• Reduced training time by nearly 50% across various datasets.
• Improved linear probing accuracy of Masked Autoencoders (MAE) by 1.3% to 3.9%.
• Demonstrated superior performance in downstream detection and segmentation tasks.

Conclusions:

• FAMT effectively extracts semantic patches, boosting model performance and training efficiency.
• The plug-and-play FAMT module significantly enhances existing MIM approaches.
• FAMT offers a promising direction for efficient and effective self-supervised visual representation learning.