Universal Stabilization for Maximum Entropy Optimization in Reinforcement Learning

In real-world decision-making tasks, it is critical for reinforcement learning (RL) methods to be both stable and robust. Maximum entropy RL methods typically generate a robust policy with entropy augmented reward. While incorporating entropy into the reward offers the benefit of exploration, it presents limited universal applicability and persistent convergence difficulties, such as suboptimal policy stabilization and unstable $Q$ value update. From optimization, we define these two issues as tremulous policy and spiky Q-function, investigating their underlying causes and relationships. Analysis with this, the maximum entropy principle leads to a spiky $Q$ -function update, which ultimately results in a tremulous policy. We thus introduce a beta-symmetric Kullback-Leibler (KL) divergence objective to mitigate such issues under the maximum entropy framework. With this objective function, the tremulous nature of the policy could be controlled with a large beta value. The spiky $Q$ -function could be avoided by annealing the entropy in the target $Q$ value, as the beta-symmetric KL divergence is an upper bound of the original reverse KL divergence. Theoretically, we prove that minimizing our new objective function results in a new policy that presents an improvement in the $Q$ value. Guaranteed by these results, we ultimately derive the optimal policy by iteratively updating the $Q$ value and policy, and we call this method max-entropy stable optimization (MeSO). Experimental results on the Mujoco and Roboschool platforms demonstrate that our algorithm maintains stability while offering better flexibility and overall performance.