Adaptive Discount Factor for Deep Reinforcement Learning in Continuing Tasks with Uncertainty

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) agents learn by maximizing discounted rewards.
• The discount factor critically impacts RL performance, especially under uncertainty.
• Constant discount factors can limit learning effectiveness in dynamic environments.

Purpose of the Study:

• To propose an adaptive rule for the discount factor in RL.
• To enhance consistent learning performance by adapting the discount factor based on the advantage function.
• To demonstrate the applicability of the adaptive rule in both on-policy and off-policy RL algorithms.

Main Methods:

• Developed an adaptive discount factor rule utilizing the advantage function.
• Integrated the adaptive rule into Proximal Policy Optimization (PPO) for on-policy learning.
• Integrated the adaptive rule into Soft Actor-Critic (SAC) for off-policy learning.
• Validated the approach on Tetris gameplay and robot manipulator motion planning.

Main Results:

• The proposed adaptive discount factor achieved comparable or superior performance to optimal constant discount factors.
• The adaptive method demonstrated effectiveness in both on-policy (PPO) and off-policy (SAC) scenarios.
• Consistent training performance was achieved across different tasks.

Conclusions:

• The adaptive discount factor rule effectively optimizes learning performance in RL.
• This method offers a robust alternative to manual discount factor tuning.
• The adaptive approach is applicable to various deep reinforcement learning problems.