Learning representations via dynamics-based behavioral similarity for deep reinforcement learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Deep reinforcement learning requires learning task-relevant representations from visual data.
• Behavioral similarity metrics group equivalent states but suffer from representation collapse due to sparse rewards.
• This limits scalability in complex applications.

Purpose of the Study:

• To propose a novel approach, Representation learning with Dynamics-based behavioral Similarity (RDS), to overcome the limitations of existing representation learning methods.
• To develop a reward-independent similarity metric that preserves behavioral discriminability for improved deep reinforcement learning.

Main Methods:

• Introduced a dynamics-driven similarity metric that eliminates reward dependency.
• Incorporated dynamic transition distances with trainable Gaussian noise to mitigate metric degradation.
• Utilized latent trajectory distances to quantify task differences and extract relevant features.

Main Results:

• RDS demonstrated superior performance over baseline methods on complex DeepMind Control, MetaWorld, and Adroit manipulation tasks.
• Achieved significant improvements of 43% and 30% over DrQ-v2 and state-of-the-art methods, respectively.
• Ablation studies confirmed the effectiveness of individual components within the RDS approach.

Conclusions:

• Representation learning with Dynamics-based behavioral Similarity (RDS) effectively addresses representation collapse in deep reinforcement learning.
• The proposed method enhances learning of task-relevant features by leveraging dynamics-based similarity, showing strong performance on challenging robotic tasks.