Enhancing Exploration in Actor-Critic Algorithms: An Approach to Incentivize Plausible Novel States

Actor-critic (AC) algorithms are model-free deep reinforcement learning techniques that have consistently demonstrated effectiveness across various domains. Enhancing exploration (action entropy) and exploitation (expected return) through more efficient sample utilization is pivotal to their success. A key strategy for a learning algorithm is to intelligently navigate the environment's state space, prioritizing the exploration of rarely visited states over frequently encountered ones. However, conventional approaches rarely quantify a novel state's utility for policy learning, which can lead to inefficient exploration. To address this, we propose an innovative approach to bolster exploration by employing an intrinsic reward based on a state's novelty and the potential benefits of exploring that state, which we term plausible novelty. Our method seamlessly integrates with off-policy AC algorithms. By incentivizing the exploration of plausibly novel states, AC algorithms can achieve substantial improvements in sample efficiency and overall training performance. Empirical results demonstrate 19% improvement in training return and 30% reduction in standard deviation, averaged across comparisons of three benchmark algorithm pairs in five different environments.