Discovering diverse solutions in deep reinforcement learning by maximizing state-action-based mutual information.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) typically produces single solutions, despite task diversity.
• Learning diverse solutions enhances few-shot adaptation capabilities.
• Existing methods using mutual information for diversity suffer from gradient estimator bias.

Purpose of the Study:

• To develop a novel reinforcement learning method for learning diverse solutions without gradient bias.
• To enable learning an infinite set of diverse solutions using continuous latent variables.
• To improve few-shot adaptation compared to existing approaches.

Main Methods:

• A policy conditioned on latent variables is trained by directly maximizing the variational lower bound of mutual information.
• This method avoids using mutual information as unsupervised rewards, mitigating bias.
• Experiments conducted on robot locomotion tasks.

Main Results:

• The proposed method successfully learns an infinite set of diverse solutions using continuous latent variables.
• Demonstrated superior few-shot adaptation performance compared to existing methods.
• Effectively addresses the bias problem inherent in prior techniques.

Conclusions:

• The novel method effectively learns diverse solutions in reinforcement learning.
• Continuous latent variables facilitate learning an infinite set of diverse policies.
• The approach offers improved robustness and adaptation capabilities for complex tasks.