The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Discount regularization is commonly used in reinforcement learning to simplify policy estimation from sparse or noisy data.
• It is generally understood to de-emphasize delayed effects in Markov Decision Processes (MDPs).

Purpose of the Study:

• To reveal an alternative perspective on discount regularization, highlighting its unintended consequences.
• To demonstrate that discount regularization is equivalent to a prior on the transition matrix.
• To propose a novel method for setting state-action-specific regularization parameters.

Main Methods:

• Developed an equivalence theorem showing discount regularization is equivalent to a prior on the transition matrix.
• Derived an explicit formula for state-action-specific regularization parameters.
• Empirically evaluated the proposed method against standard discount regularization using simulations and a medical cancer simulator.

Main Results:

• Demonstrated that discount regularization functions as a prior with stronger regularization on state-action pairs with more data.
• Showcased that this leads to suboptimal performance when transition matrices are estimated from uneven datasets.
• The proposed state-action-specific method significantly improves policy estimation accuracy.

Conclusions:

• Discount regularization has unintended consequences, acting as a data-dependent prior.
• A state-action-specific regularization approach remedies the shortcomings of global discount regularization.
• The proposed method offers improved policy estimation in reinforcement learning, especially with uneven data.