Optimistic reinforcement learning by forward Kullback-Leibler divergence optimization.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Traditional reinforcement learning (RL) optimizes policies indirectly to maximize returns.
• Recent work interprets RL optimization with explicit consideration of optimality as a stochastic variable.
• Existing methods often use reverse Kullback-Leibler (KL) divergence for optimization.

Purpose of the Study:

• To propose a new interpretation of RL optimization using KL divergence.
• To derive a novel optimization method based on forward KL divergence.
• To investigate the impact of optimism in RL for improved learning.

Main Methods:

• Formulated traditional RL learning laws as optimization problems with reverse KL divergence.
• Derived new optimization problems utilizing forward KL divergence, addressing KL divergence asymmetry.
• Introduced an optimistic RL approach controlled by a hyperparameter derived from uncertainty.

Main Results:

• The derived forward KL divergence optimization problems are interpretable as optimistic RL.
• Optimism, controlled by a hyperparameter, was found to accelerate learning and increase rewards.
• Integration with prioritized experience replay and eligibility traces further enhanced learning speed.

Conclusions:

• A novel optimistic RL method based on forward KL divergence was successfully derived.
• Moderate optimism demonstrably accelerates learning and improves reward acquisition in simulations.
• The proposed method showed superior performance compared to state-of-the-art RL techniques in realistic robotic simulations.