Heuristic Attention Representation Learning for Self-Supervised Pretraining.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Self-supervised learning (SSL) excels at representation learning by maximizing similarity across augmented image views.
• A key challenge in SSL is semantic feature inconsistency across views due to random cropping.

Purpose of the Study:

• To introduce Heuristic Attention Representation Learning (HARL), a novel SSL framework.
• To address semantic feature inconsistencies in SSL by focusing on object-level representations.

Main Methods:

• HARL utilizes a joint embedding architecture with two neural networks trained on augmented image views.
• It incorporates prior visual object-level attention by generating heuristic mask proposals.
• The framework maximizes object-level embedding similarity, rather than whole-image similarity.

Main Results:

• HARL extracts high-quality semantic representations, outperforming existing SSL baselines.
• Achieved +1.3% on ImageNet semi-supervised learning and +0.9% AP50 on COCO object detection.
• Demonstrated effective heuristic mask proposal generation techniques for natural images.

Conclusions:

• HARL offers a robust approach to self-supervised representation learning by incorporating object-level attention.
• The method enhances performance on downstream tasks like image classification and object detection.
• Code availability in TensorFlow and PyTorch facilitates broader adoption and research.