Relative importance sampling for off-policy actor-critic in deep reinforcement learning.

Area of Science:

• Reinforcement Learning
• Machine Learning
• Artificial Intelligence

Background:

• Off-policy reinforcement learning (RL) suffers from instability compared to on-policy methods.
• Distributional mismatch between target and behavior policies causes high variance and instability.
• Importance sampling (IS) reduces distributional mismatch but introduces high variance, especially in sequential tasks.

Purpose of the Study:

• To introduce Relative Importance Sampling (RIS) as a method to mitigate variance and stabilize off-policy RL.
• To develop the first model-free RIS off-policy actor-critic (RIS-off-PAC) algorithms.
• To investigate the effect of the smoothness parameter [Formula: see text] on variance control.

Main Methods:

• Proposed Relative Importance Sampling (RIS) to smooth importance sampling and reduce variance.
• Developed model-free RIS off-policy actor-critic (RIS-off-PAC) algorithms using deep neural networks.
• Trained actor and critic networks using behavior policy action values in the reward function.

Main Results:

• The proposed RIS-off-PAC algorithms demonstrated improved stability and performance over state-of-the-art RL benchmarks.
• Performance was evaluated on OpenAI Gym challenges and synthetic datasets.
• The smoothness parameter [Formula: see text] in RIS effectively controls variance.

Conclusions:

• Relative Importance Sampling (RIS) offers a promising approach to stabilize and enhance off-policy reinforcement learning.
• The developed RIS-off-PAC algorithms provide a robust and effective solution for complex RL tasks.
• This work lays the foundation for future research in variance reduction techniques for RL.