Entropic Regularization of Markov Decision Processes.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Optimal feedback controllers for Markov decision processes (MDPs) are typically synthesized via value or policy iteration.
• Learning agents interacting with unknown environments require regularization to prevent divergence to unsafe states.
• Previous methods used Kullback-Leibler (KL) divergence to stabilize policy improvement, but a broader approach is needed.

Purpose of the Study:

• To generalize policy optimization using a wider class of divergences, specifically f-divergences and alpha-divergences.
• To provide a unified perspective on actor-critic architectures through an entropic proximal policy optimization framework.
• To analyze the impact of different divergence functions on reinforcement learning dynamics and performance.

Main Methods:

• The study extends policy improvement steps to utilize a family of f-divergences, including alpha-divergences.
• A dual objective for policy evaluation is derived, unifying compatible actor-critic architectures.
• Asymptotic analysis is performed on solutions derived from alpha-divergences for various alpha values.

Main Results:

• The framework unifies existing methods, showing least-squares value function estimation with advantage-weighted maximum likelihood policy improvement corresponds to the Pearson chi-2 divergence.
• Different choices of the penalty-generating function f lead to various actor-critic pairs.
• The impact of selecting specific divergence functions on reinforcement learning problems is demonstrated.

Conclusions:

• The proposed f-divergence framework offers a more flexible and stable approach to reinforcement learning policy optimization.
• The unified perspective clarifies the relationship between different actor-critic algorithms and divergence measures.
• The choice of divergence function significantly influences the behavior and performance of reinforcement learning agents.