Robust Losses for Learning Value Functions.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Optimization

Background:

• Reinforcement learning (RL) often uses mean squared Bellman error, which is sensitive to outliers.
• Outliers cause skewed solutions and high-variance gradients, necessitating clipping or rescaling strategies.
• Current RL methods with these strategies use semi-gradient rules, not minimizing a defined loss.

Purpose of the Study:

• Reformulate Bellman errors using robust losses like Huber and Absolute Bellman error.
• Develop sound gradient-based algorithms for online, off-policy prediction and control.
• Analyze the benefits of robust losses over mean squared Bellman error.

Main Methods:

• Reformulated squared Bellman errors as a saddlepoint optimization problem.
• Derived gradient-based algorithms for Huber Bellman error and Absolute Bellman error.
• Formalized robust loss functions and analyzed their properties.

Main Results:

• Proposed saddlepoint reformulations for robust Bellman errors.
• Derived gradient-based algorithms for prediction and control settings.
• Characterized solutions, showing advantages over mean squared Bellman error in specific scenarios.

Conclusions:

• Robust Bellman error algorithms demonstrate improved stability in RL.
• The proposed methods are less sensitive to meta-parameters compared to traditional approaches.
• This work provides a theoretical foundation and practical algorithms for robust value function learning.