Variance Reduced Domain Randomization for Reinforcement Learning With Policy Gradient.

Area of Science:

• Robotics
• Machine Learning
• Artificial Intelligence

Background:

• Domain randomization (DR) is crucial for improving the generalization capabilities of deep reinforcement learning (DRL) policies by introducing environmental diversity.
• However, DR exacerbates policy gradient variance due to increased environmental variability, hindering sample efficiency in DRL training.
• Standard baselines in DR often fail to sufficiently mitigate this high variance, leading to suboptimal training performance.

Purpose of the Study:

• To theoretically derive a bias-free, state/environment-dependent optimal baseline for domain randomization (DR) in DRL.
• To propose a novel variance-reduced domain randomization (VRDR) method that balances variance reduction with computational feasibility.
• To demonstrate VRDR's effectiveness in accelerating convergence and enhancing policy stability and performance in robotic control.

Main Methods:

• Theoretical derivation of an optimal baseline for DR that accounts for both state and environment dependencies.
• Development of the variance-reduced domain randomization (VRDR) approach, which estimates baselines within environment subspaces.
• Empirical evaluation of VRDR on six robot control tasks featuring randomized dynamics.

Main Results:

• The proposed optimal baseline theoretically guarantees further variance reduction compared to standard constant and state-dependent baselines in DR.
• VRDR demonstrates significant acceleration in policy training convergence compared to state-dependent baselines.
• Empirical results show VRDR consistently achieves superior final policies with improved training stability across multiple robotic tasks.

Conclusions:

• The novel VRDR approach effectively reduces variance in policy gradient estimation for DR.
• VRDR offers a practical and theoretically grounded method for enhancing DRL training efficiency and final policy quality in randomized environments.
• This work provides a significant advancement for applying DRL to real-world robotic systems requiring robust generalization.