Robust Reinforcement Learning via Leveraging Historically Optimal Policy With Regulation of Performance

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Existing adversarial training methods in Reinforcement Learning (RL) exhibit limited robustness against novel attacks.
• Current RL approaches often rely on simple trial-and-error, leaving them vulnerable to sophisticated adversarial perturbations.

Purpose of the Study:

• To introduce a novel approach, Robust RL via Leveraging Historically Optimal Policy with Regulation of Performance (HORP), to enhance policy robustness in RL.
• To develop an RL agent capable of generalized defense against diverse state attacks.

Main Methods:

• HORP utilizes the historically optimal policy to guide policy optimization and generate diverse adversarial perturbations.
• A guidance value function is constructed by considering value gaps and policy distribution divergence for prioritized learning.
• An adaptive performance-aware optimization mechanism and dynamic perturbation entropy modulation are employed to improve robustness.

Main Results:

• HORP demonstrates superior performance in both natural performance and robustness against various state attacks compared to existing methods.
• The approach effectively focuses learning on promising action spaces and prevents agent deviation from optimal performance.
• Experiments confirm enhanced generalized defensive capabilities through controlled uncertainty injection.

Conclusions:

• HORP offers a significant advancement in adversarial robustness for Reinforcement Learning.
• The proposed method provides a more reliable and robust RL agent, capable of withstanding novel and diverse attacks.
• HORP represents a promising direction for developing resilient AI systems in security-critical applications.