Modeling Bellman-error with logistic distribution with applications in reinforcement learning.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Neuroscience

Background:

• Optimizing the Bellman error is crucial in Reinforcement Learning (RL), with mean-squared error (MSELoss) being the conventional choice.
• The assumption of Gaussian distribution for Bellman errors may not fully capture the complexities of RL training dynamics.

Purpose of the Study:

• To investigate the actual distribution of Bellman errors during RL training.
• To propose and validate the use of a Logistic loss function (LLoss) as a superior alternative to MSELoss.
• To explore the theoretical connections between Bellman error distribution and RL techniques like proportional reward scaling.

Main Methods:

• Analyzing Bellman error distributions in RL training across various environments.
• Implementing and evaluating a Logistic maximum likelihood function (LLoss) against MSELoss in baseline RL algorithms.
• Utilizing Kolmogorov-Smirnov tests to statistically compare distribution fits.
• Applying Bias-Variance decomposition to analyze the sample-accuracy trade-off for Logistic distribution approximation.

Main Results:

• Bellman errors in RL training empirically follow a Logistic distribution, not a Gaussian one.
• Replacing MSELoss with LLoss in RL algorithms consistently yields improved performance.
• Kolmogorov-Smirnov tests confirm the Logistic distribution as a more accurate fit for Bellman errors.
• A novel theoretical link between Bellman error distribution and proportional reward scaling is established.

Conclusions:

• The Logistic distribution provides a more accurate model for Bellman errors in RL training.
• Adopting LLoss enhances the performance of various RL algorithms compared to MSELoss.
• This research offers foundational insights for distribution-based optimization in RL and future advancements.