Policy Gradient From Demonstration and Curiosity.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) agents learn complex behaviors from task abstractions.
• Exploration and reward shaping are challenging in RL, especially with sparse extrinsic feedback.
• Existing methods often require numerous high-quality expert demonstrations, which are difficult to obtain.

Purpose of the Study:

• To propose an integrated policy gradient algorithm for enhanced exploration and intrinsic reward learning.
• To address the challenge of learning from a limited number of expert demonstrations in RL.
• To improve agent performance in environments with sparse reward signals.

Main Methods:

• Developed an integrated policy gradient algorithm.
• Reformulated the reward function with Jensen-Shannon divergence between policy and expert demonstrations.
• Incorporated an agent's environmental uncertainty estimation into the reward function.
• Evaluated the algorithm on simulated tasks with sparse rewards and limited demonstrations.

Main Results:

• Demonstrated superior exploration efficiency across all tested tasks.
• Achieved high average returns in environments with sparse extrinsic rewards.
• Showcased the agent's ability to imitate expert behavior effectively.
• Validated the algorithm's performance with limited expert trajectories.

Conclusions:

• The proposed algorithm effectively boosts exploration and intrinsic reward learning in RL.
• Limited expert demonstrations can be leveraged for improved agent performance.
• The method balances imitation of expert behavior with maintaining high returns.