Forward and inverse reinforcement learning sharing network weights and hyperparameters

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Imitation learning aims to train agents by observing expert demonstrations.
• Model-free methods often struggle with sample efficiency and defining appropriate reward functions.
• Reinforcement learning (RL) can be sample-inefficient and requires careful reward engineering.

Purpose of the Study:

• To propose a novel model-free imitation learning algorithm, Entropy-Regularized Imitation Learning (ERIL).
• To enhance sample efficiency and learning performance compared to existing imitation learning methods.
• To apply ERIL to real-world robotic tasks and human behavior analysis.

Main Methods:

• ERIL minimizes the reverse Kullback-Leibler (KL) divergence within an entropy-regularized Markov decision process framework.
• It combines forward and inverse reinforcement learning (RL) using two binary discriminators.
• The second discriminator's hyperparameters are shared with forward RL for controlled learning.

Main Results:

• ERIL demonstrated superior sample efficiency over baseline methods in MuJoCo simulations and robotic reaching tasks.
• The method successfully applied to human pole-balancing behaviors, revealing insights into goal achievement strategies.
• Minimizing reverse KL divergence was shown to be equivalent to finding an optimal policy.

Conclusions:

• ERIL offers a more sample-efficient and effective approach to model-free imitation learning.
• The method's ability to estimate reward functions provides valuable insights into agent behavior.
• ERIL shows promise for applications in robotics, human behavior modeling, and reinforcement learning research.