Oracle-Guided Masked Contrastive Reinforcement Learning for Visuomotor Policies.

Area of Science:

• Robotics
• Machine Learning
• Computer Vision

Background:

• Visuomotor policy learning often struggles with sample efficiency and sim-to-real gaps due to high-dimensional inputs.
• Current reinforcement learning methods face challenges mapping complex visual data to precise actions.

Purpose of the Study:

• To introduce a novel framework, Oracle-Guided Masked Contrastive Reinforcement Learning (OMC-RL), to enhance sample efficiency and performance in visuomotor policy learning.
• To address limitations in current approaches for learning policies from visual observations.

Main Methods:

• OMC-RL decouples learning into representation learning (upstream) and policy learning (downstream).
• The upstream stage uses a masked Transformer with temporal modeling and contrastive learning for representation extraction.
• The downstream stage employs an oracle teacher policy for supervised guidance, gradually reducing as training progresses.

Main Results:

• OMC-RL demonstrated superior sample efficiency and asymptotic performance in both simulated and real-world experiments.
• The framework showed improved generalization capabilities across diverse and perceptually complex environments.
• The learned encoder effectively extracts temporally-aware and task-relevant visual representations.

Conclusions:

• OMC-RL offers a significant advancement in visuomotor policy learning, overcoming key challenges in sample efficiency and sim-to-real transfer.
• The proposed two-stage approach effectively leverages representation learning and guided policy training for robust performance.
• This framework holds promise for developing more capable and generalizable robotic agents.