Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Efficient exploration is crucial in reinforcement learning (RL), particularly with sparse or absent extrinsic rewards.
• Intrinsic motivation methods show promise but struggle with multimodal and stochastic environmental dynamics.
• Existing environment model-based exploration approaches have limitations in complex scenarios.

Purpose of the Study:

• To develop a novel variational dynamic model capable of modeling multimodality and stochasticity in RL environments.
• To enhance agent exploration capabilities by improving the understanding of environmental dynamics.
• To enable self-supervised exploration without relying on extrinsic rewards.

Main Methods:

• Proposed a variational dynamic model utilizing conditional variational inference to capture environmental multimodality and stochasticity.
• Modeled environmental state-action transitions as a conditional generative process incorporating latent variables.
• Derived an upper bound of the negative log likelihood of environmental transitions to serve as an intrinsic reward signal.

Main Results:

• The proposed method demonstrated superior performance in exploration compared to state-of-the-art approaches.
• Effectiveness was validated across various image-based simulation tasks and a real-world robotic manipulation task.
• The model successfully addressed challenges posed by multimodal and stochastic dynamics.

Conclusions:

• The variational dynamic model offers a robust solution for efficient exploration in challenging RL environments.
• Conditional variational inference and self-supervised exploration are effective strategies for overcoming sparse reward limitations.
• The approach advances the field of model-based reinforcement learning and robotic control.