Bootstrap Masked Visual Modeling via Hard Patch Mining.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Masked visual modeling is crucial for learning generalizable representations.
• Current methods focus on predicting masked content, with performance linked to masking strategies.
• The model's role as a 'teacher' in generating challenging problems is underexplored.

Purpose of the Study:

• To propose a new method, Hard Patch Mining (U²PL+), that empowers models to act as teachers by identifying hard-to-reconstruct patches.
• To improve representation learning in masked visual modeling by focusing on self-generated challenging tasks.

Main Methods:

• Introduced Hard Patch Mining (U²PL+) to predict patch-wise reconstruction losses.
• Employed an auxiliary loss predictor trained with a relative objective to avoid overfitting.
• Implemented an easy-to-hard mask strategy to guide the training process.

Main Results:

• U²PL+ demonstrated significant improvements on both image and video benchmarks.
• The auxiliary loss prediction objective alone enhanced representation quality.
• The efficacy of identifying hard-to-reconstruct areas was validated.

Conclusions:

• Hard Patch Mining (U²PL+) offers a more effective approach to masked visual modeling by shifting focus from problem-solving to problem generation.
• This method enhances the learning of generalizable representations by leveraging the model's ability to identify and mask difficult patches.