On Practical Robust Reinforcement Learning: Adjacent Uncertainty Set and Double-Agent Algorithm.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Robust reinforcement learning (RRL) optimizes policies against worst-case scenarios within an uncertainty set of Markov decision processes (MDPs).
• Current RRL methods are limited to tabular settings, hindering application in complex, continuous state/action spaces.
• Real-world deployment faces challenges due to potential mismatches between training simulations and actual environments.

Purpose of the Study:

• To extend robust reinforcement learning algorithms to continuous state and action spaces.
• To develop a novel RRL approach that guarantees improved robustness compared to existing methods.
• To address the limitations of current RRL techniques in handling complex environments.

Main Methods:

• Constructed an elaborated uncertainty set of plausible perturbed MDPs.
• Proposed adjacent robust Q-learning (ARQ-Learning) for tabular settings with finite-time error bounds.
• Introduced a dual-agent approach with a pessimistic agent to enable extension to continuous spaces.

Main Results:

• ARQ-Learning demonstrates convergence comparable to Q-learning and Robust-Q, with enhanced robustness guarantees.
• The novel dual-agent method successfully extends model-free RRL to continuous state/action spaces for the first time.
• Experimental validation confirmed the effectiveness of the proposed algorithms in continuous environments.

Conclusions:

• The developed RRL algorithms effectively handle continuous state/action spaces, a significant advancement over prior work.
• The dual-agent approach provides a robust solution for optimizing policies in complex, uncertain environments.
• This research opens new avenues for applying robust reinforcement learning in real-world applications with continuous dynamics.