Realistic Actor-Critic: A framework for balance between value overestimation and underestimation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Value approximation bias in reinforcement learning leads to suboptimal policies and hinders effective exploration-exploitation decisions.
• Existing algorithms struggle to provide a clear understanding of value bias impact and methods for stable, efficient exploration.

Purpose of the Study:

• To clarify the impact of value bias on reinforcement learning performance.
• To develop a method for efficient exploration with stable updates.
• To enhance sample efficiency in reinforcement learning algorithms.

Main Methods:

• Designed a simple episodic tabular MDP to study value underestimation and overestimation in actor-critic methods.
• Proposed the Realistic Actor-Critic (RAC) framework using Universal Value Function Approximators (UVFA).
• RAC learns policies with different value confidence bounds within a single neural network to manage under/overestimation trade-offs.

Main Results:

• Identified that fixed hyperparameter settings can cause agents to over-explore low-value states.
• RAC enables directed exploration using upper bounds and avoids overestimation with lower bounds.
• Achieved 10x sample efficiency and 25% performance improvement over Soft Actor-Critic in the Humanoid environment.

Conclusions:

• Provides insights into the exploration-exploitation trade-off by analyzing policy access to low-value states under varying confidence bounds.
• Proposes RAC as a unified framework to enhance sample efficiency in continuous control domains when combined with current actor-critic methods.