Adaptive importance sampling for value function approximation in off-policy reinforcement learning.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Statistics

Background:

• Off-policy reinforcement learning (RL) utilizes data from different policies for efficient learning.
• Importance sampling corrects bias in value function estimation but can increase variance.
• Existing methods often struggle with estimator variance, leading to unstable performance.

Purpose of the Study:

• To develop a more stable and efficient off-policy reinforcement learning method.
• To actively control the bias-variance trade-off in value function estimation.
• To introduce an adaptive importance sampling technique for improved RL performance.

Main Methods:

• Proposed an adaptive importance sampling (AIS) technique to manage bias-variance.
• Developed a cross-validation-based method for optimal parameter selection in AIS.
• Evaluated the approach through simulation studies.

Main Results:

• The proposed adaptive importance sampling method demonstrated effective control over bias-variance trade-offs.
• Optimal parameter determination using cross-validation led to improved estimator stability.
• Simulations confirmed the enhanced performance and stability of the new approach.

Conclusions:

• Adaptive importance sampling offers a robust solution for stabilizing off-policy reinforcement learning.
• The bias-variance control mechanism is crucial for reliable performance in RL.
• This work provides a practical method for improving data efficiency and stability in RL algorithms.