Assessing Transferability From Simulation to Reality for Reinforcement Learning.

Area of Science:

• Robotics
• Machine Learning
• Control Theory

Background:

• Learning robot control policies in simulation offers efficiency and safety benefits over real-world experiments.
• Direct transfer of policies from simulation to reality is hindered by 'Simulation Optimization Bias' (SOB), where policies exploit simulator inaccuracies and risk damaging robots.

Purpose of the Study:

• To develop a method for training robot control policies in simulation that are directly transferable to real-world systems.
• To address the challenge of Simulation Optimization Bias (SOB) in simulation-based reinforcement learning for robotics.

Main Methods:

• Domain randomization was applied by varying physics simulation parameters during policy learning.
• A novel algorithm, Simulation-based Policy Optimization with Transferability Assessment (SPOTA), was proposed.
• SPOTA incorporates an estimator of SOB to define a training stopping criterion, quantifying overfitting to simulated domains.

Main Results:

• The SPOTA algorithm successfully learned control policies exclusively within a randomized simulation environment.
• Experimental validation on two nonlinear systems demonstrated the direct applicability of learned policies to real robots without further training.
• The SOB estimator effectively quantified over-fitting, guiding the training process.

Conclusions:

• SPOTA enables robust robot control policy learning from simulation, overcoming the simulation-to-reality gap.
• The method mitigates risks associated with Simulation Optimization Bias, enhancing the reliability of simulated training.
• This approach facilitates faster, cheaper, and safer robot development by reducing reliance on physical prototypes.