Variance Regularized Counterfactual Risk Minimizationvia Variational Divergence Minimization.

Area of Science:

• Machine Learning
• Reinforcement Learning
• Artificial Intelligence

Background:

• Off-policy learning evaluates and improves policies from historical data, crucial due to the cost and impact of on-policy evaluation.
• A key challenge in off-policy learning is developing low-variance counterfactual estimators to minimize generalization error.

Purpose of the Study:

• To present a novel counterfactual learning principle for off-policy learning with bandit feedback.
• To regularize generalization error by minimizing distribution divergence between logging and new policies.
• To eliminate the need for sample variance computation in training.

Main Methods:

• Inspired by learning bounds for importance sampling.
• Developed a new principle for off-policy learning with bandit feedback.
• Utilized variational divergence minimization for end-to-end training with neural network policies.

Main Results:

• The proposed method effectively regularizes generalization error.
• Demonstrated significant performance improvements over conventional baseline algorithms.
• Achieved results consistent with theoretical findings.

Conclusions:

• The new counterfactual learning principle offers an effective approach for off-policy learning.
• Variational divergence minimization with neural networks provides a powerful tool for improving policy evaluation and learning.
• This work advances the field of off-policy learning by addressing key challenges in variance reduction and generalization.