Visual reinforcement learning via sequential consistency preserved policy contrast from optimal transport view

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Visual reinforcement learning (RL) struggles with data inefficiency in complex tasks.
• Current methods use representation learning with pixel-contrast objectives, which may not capture essential decision-making information.
• These representations can potentially hinder policy learning.

Purpose of the Study:

• To propose a novel methodology, CoCo (sequential Consistency preserved policy Contrast), to address data inefficiency in visual RL.
• To capture invariant policy-based discriminative features by contrasting policies across multiple distorted observation views.
• To model policy contrast as an optimal transport problem and align policy distributions during contrastive learning.

Main Methods:

• CoCo performs policy contrast across multiple distorted views of observations.
• Policy contrast is modeled as an optimal transport problem, aligning policy distributions.
• An inverse consistency-weighting mechanism accentuates view differences while preserving semantic integrity.

Main Results:

• CoCo captures essential policy-based discriminative features, overcoming limitations of observation-contrast methods.
• Theoretical optimality is established through information-theoretic analysis.
• CoCo consistently outperforms existing approaches on data efficiency benchmarks.

Conclusions:

• CoCo offers a superior approach to visual reinforcement learning by focusing on policy-invariant features.
• The method effectively addresses data inefficiency and improves decision-making in complex scenarios.
• CoCo represents a significant advancement in representation learning for RL.