Partial Consistency for Stabilizing Undiscounted Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Undiscounted return optimization in reinforcement learning (RL) often leads to training instability.
• Existing studies lack in-depth analysis of the causes behind this instability.
• Transient traps, stemming from inconsistent action selection, are identified as a key issue.

Purpose of the Study:

• To analyze the causes of training instability in RL when optimizing undiscounted returns.
• To propose a novel method for balancing exploration and stability.
• To ensure consistent action selection in specific states to mitigate instability.

Main Methods:

• Analysis of training instability from a value estimation perspective.
• Introduction of Last-Visit Sampling (LVS), a novel action sampling method.
• Decomposition of state-action value into last-visit (LV) and revisit values.

Main Results:

• The study proves that LVS can eliminate transient traps while preserving optimality.
• Empirical results demonstrate that LVS stabilizes training processes across five diverse tasks.
• Tasks included vision-based navigation and manipulation, showcasing broad applicability.

Conclusions:

• Last-visit sampling (LVS) effectively addresses training instability in reinforcement learning.
• The method ensures consistent action selection, crucial for stable value estimation.
• LVS offers a promising approach for real-world RL applications requiring reliable performance.