Perceptual Inductive Bias Is What You Need Before Contrastive Learning

Area of Science:

• Computer Vision
• Cognitive Science
• Machine Learning

Background:

• Human perception follows a multi-stage process, prioritizing boundary and surface properties before semantic understanding.
• Current contrastive representation learning bypasses these stages, leading to slower convergence and texture bias.
• This study bridges the gap between computational vision and cognitive theories of perception.

Purpose of the Study:

• To investigate the benefits of incorporating a multi-stage perceptual approach into contrastive representation learning.
• To improve convergence speed, representation quality, and robustness in visual AI models.
• To leverage inductive biases from human vision systems for more efficient AI.

Main Methods:

• Implemented a novel pretraining stage based on David Marr's theory of perception.
• Focused on constructing boundary and surface-level representations before semantic object learning.
• Utilized ResNet18 architecture for evaluating the proposed method.

Main Results:

• Achieved 2x faster convergence on ResNet18 compared to standard contrastive learning.
• Demonstrated improved performance in semantic segmentation, depth estimation, and object recognition.
• Showcased enhanced robustness and out-of-distribution capabilities of the learned representations.

Conclusions:

• Integrating multi-stage perceptual processing significantly enhances representation learning in AI.
• The proposed method offers a more efficient and robust approach to visual AI, inspired by human vision.
• This work paves the way for developing AI systems with more human-like visual understanding.