Adversary Agnostic Robust Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Deep reinforcement learning (DRL) policies are vulnerable to adversarial perturbations during testing, impacting their reliability.
• Existing methods to improve DRL robustness often require explicit adversarial training data, increasing costs and potentially introducing new vulnerabilities.

Purpose of the Study:

• To propose a novel, adversary-agnostic DRL paradigm that enhances policy robustness without prior knowledge of specific attacks.
• To theoretically and empirically validate a new policy distillation (PD) loss for improving adversarial robustness in DRL.

Main Methods:

• Developed a new policy distillation (PD) loss comprising a prescription gap maximization (PGM) loss and a Jacobian regularization (JR) loss.
• The PGM loss maximizes the likelihood of the teacher policy's action and the entropy of other actions.
• The JR loss minimizes the gradient magnitude of the policy with respect to the input state, ensuring robustness.

Main Results:

• Theoretical analysis confirmed that the proposed distillation loss increases the prescription gap, thereby enhancing adversarial robustness.
• Experimental results on five Atari games demonstrated the superiority of the proposed method over state-of-the-art baselines.
• The adversary-agnostic approach achieved significant robustness improvements without explicit adversarial training.

Conclusions:

• The proposed adversary-agnostic DRL paradigm effectively enhances policy robustness against unseen perturbations.
• The novel PD loss, combining PGM and JR, provides a theoretically grounded and empirically validated method for improving DRL security.
• This approach offers a more efficient and versatile solution for robust DRL compared to methods requiring explicit adversarial training.