Risk-Sensitive Reinforcement Learning With Exponential Criteria.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) demonstrates success but suffers from sensitivity to noise and parameter perturbations.
• This sensitivity leads to high reward variability across different episodes and environments.
• Risk-sensitive RL methods are crucial for enhancing robustness and sample efficiency.

Purpose of the Study:

• To define robust RL policies and formulate a risk-sensitive RL problem to approximate them.
• To address the limitations of traditional RL by introducing robustness and improving sample efficiency.
• To develop novel risk-sensitive algorithms for improved RL performance.

Main Methods:

• Formulated a risk-sensitive RL problem using exponential criteria for optimization.
• Studied a model-free risk-sensitive variation of the Monte Carlo policy gradient algorithm.
• Introduced a novel risk-sensitive online Actor-Critic algorithm utilizing a multiplicative Bellman equation and stochastic approximation.

Main Results:

• Analytical results indicate that exponential criteria generalize common regularization techniques.
• The proposed methods demonstrate improved sample efficiency compared to existing approaches.
• The methods introduce robustness against perturbations in model parameters and the environment.

Conclusions:

• The developed risk-sensitive RL algorithms offer a robust and sample-efficient alternative to traditional methods.
• The use of exponential criteria provides a generalized framework for risk-sensitive RL.
• Simulated experiments validate the implementation, performance, and robustness of the proposed algorithms.