Online Reinforcement Learning Using a Probability Density Estimation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Online, incremental reinforcement learning faces challenges with biased sampling and nonstationarity.
• Biased sampling leads to over-representation of certain regions, distorting function approximation.
• Nonstationarity arises from recursive estimation in temporal difference methods, varying locally.

Purpose of the Study:

• To propose a novel method for function approximation in online reinforcement learning.
• To address the dual challenges of biased sampling and local nonstationarity.
• To improve the accuracy and robustness of reinforcement learning approximations.

Main Methods:

• Utilizing a Gaussian mixture model to estimate sample probability density.
• Implementing a locally adaptive forgetting factor dependent on sample density to handle nonstationarity.
• Modulating forgetting factors for mixture components based on new information to mitigate biased sampling.

Main Results:

• The proposed method effectively handles nonstationarity by adjusting forgetting factors based on local sample density.
• Biased sampling issues are mitigated by adapting forgetting factors to new data, preventing distortions in under-sampled areas.
• Improved function approximation accuracy and stability in challenging reinforcement learning scenarios.

Conclusions:

• Gaussian mixture models offer a robust framework for addressing biased sampling and nonstationarity in reinforcement learning.
• Locally adaptive forgetting factors are crucial for accurate function approximation in dynamic and unevenly sampled environments.
• This approach enhances the reliability of online, incremental reinforcement learning algorithms.