Enhancing Exploration in Actor-Critic Algorithms: An Approach to Incentivize Plausible Novel States.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Actor-critic (AC) algorithms are effective model-free deep reinforcement learning methods.
• Efficient sample utilization for exploration and exploitation is crucial for AC algorithm success.
• Current methods often fail to quantify the utility of novel states for policy learning, leading to inefficient exploration.

Purpose of the Study:

• To introduce an intrinsic reward mechanism, termed plausible novelty, to enhance exploration in AC algorithms.
• To improve sample efficiency and overall training performance by incentivizing the exploration of states with high potential learning benefits.

Main Methods:

• Developed an intrinsic reward signal based on state novelty and its potential utility for policy learning.
• Integrated the plausible novelty reward into off-policy actor-critic algorithms.
• Evaluated the proposed method on benchmark deep reinforcement learning environments.

Main Results:

• The proposed method demonstrated substantial improvements in sample efficiency.
• Achieved an average of 19% improvement in training return across multiple environments and algorithms.
• Showcased a 30% reduction in standard deviation, indicating more stable training.

Conclusions:

• Plausible novelty effectively enhances exploration in actor-critic algorithms.
• The approach leads to significant gains in sample efficiency and training performance.
• This method offers a promising direction for advancing deep reinforcement learning techniques.