Universal Stabilization for Maximum Entropy Optimization in Reinforcement Learning.

Area of Science:

• Reinforcement Learning
• Machine Learning Theory
• Decision Making

Background:

• Maximum entropy reinforcement learning (RL) methods enhance robustness but suffer from convergence difficulties.
• Issues include suboptimal policy stabilization and unstable Q-value updates, termed 'tremulous policy' and 'spiky Q-function'.

Purpose of the Study:

• To address the stability and convergence challenges in maximum entropy RL.
• To introduce a novel objective function that mitigates tremulous policies and spiky Q-functions.

Main Methods:

• Introduced a beta-symmetric Kullback-Leibler (KL) divergence objective within the maximum entropy framework.
• Developed a method called max-entropy stable optimization (MeSO) involving iterative Q-value and policy updates.
• Annealed entropy in the target Q-value to avoid spiky Q-functions.

Main Results:

• The beta-symmetric KL divergence objective controls policy tremulousness with a large beta value.
• Minimizing the new objective function theoretically improves the Q-value.
• MeSO demonstrated stability, flexibility, and enhanced overall performance in experiments.

Conclusions:

• The proposed beta-symmetric KL divergence objective effectively stabilizes maximum entropy RL.
• MeSO offers a robust and performant alternative for real-world decision-making tasks.
• The method improves upon existing maximum entropy RL approaches.