Fast Value Tracking for Deep Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Reinforcement learning (RL) agents interact with environments for sequential decision-making.
• Current RL algorithms often overlook environmental stochasticity and uncertainty quantification.
• Static models focus on point estimates, neglecting dynamic interactions.

Purpose of the Study:

• Introduce a novel, scalable sampling algorithm for deep reinforcement learning.
• Address limitations in existing RL methods regarding uncertainty quantification.
• Develop a method to quantify and monitor uncertainties during RL training.

Main Methods:

• Leverage the Kalman filtering paradigm.
• Introduce the Langevinized Kalman Temporal-Difference (LKTD) algorithm.
• Utilize Stochastic Gradient Markov Chain Monte Carlo (SGMCMC) for posterior sampling of neural network parameters.

Main Results:

• Prove convergence of LKTD posterior samples to a stationary distribution under mild conditions.
• Enable quantification of uncertainties in value functions and model parameters.
• Allow monitoring of uncertainties during policy updates in deep reinforcement learning.

Conclusions:

• The LKTD algorithm provides a robust approach for uncertainty quantification in RL.
• LKTD facilitates more adaptable and reliable reinforcement learning systems.
• This method enhances the understanding and management of uncertainty in agent-environment interactions.