Improving robustness by action correction via multi-step maximum risk estimation.

Area of Science:

• Artificial Intelligence
• Control Systems Engineering
• Machine Learning

Background:

• Ensuring control system robustness against external uncertainties is crucial for stability.
• Existing adversarial learning methods often use fixed perturbations, leading to vulnerabilities against dynamic or foresighted attacks.

Purpose of the Study:

• To develop a novel algorithm, MMRAR-RL, that optimizes reinforcement learning policies for enhanced robustness against adversarial perturbations.
• To address the limitations of greedy, fixed-strength adversarial attacks in reinforcement learning.

Main Methods:

• MMRAR-RL employs a two-stage approach: risk assessment and policy improvement.
• Risk assessment involves an adversary adaptively allocating budgets for multi-step perturbations based on agent trajectories.
• Policy improvement utilizes a maximal risk Bellman operator to estimate and mitigate policy risk.

Main Results:

• MMRAR-RL effectively estimates maximum policy risk under dynamic attack budgets.
• The algorithm demonstrates state-of-the-art performance in adversarial conditions.
• Experiments confirm MMRAR-RL's ability to tolerate significant action perturbations.

Conclusions:

• MMRAR-RL provides a robust solution for deep reinforcement learning in uncertain environments.
• The proposed method enhances stability and performance against sophisticated adversarial attacks.