Incremental model-based reinforcement learning with model constraint.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Model-based reinforcement learning (RL) relies on environment models learned from limited data for policy optimization.
• Existing methods face performance limitations due to incomplete incremental updates in both policy and model estimations.
• This gap hinders the reliable performance improvement of model-based RL algorithms.

Purpose of the Study:

• To propose a novel incremental update scheme for model-based RL.
• To guarantee simultaneous incremental updates for both the environment model and the policy.
• To ensure non-decreasing policy performance in the real environment.

Main Methods:

• Developed an incremental model-based RL update scheme with guaranteed incremental model and policy constraints.
• Established a theoretical performance bound linking the real environment and the learned model.
• Introduced the Incremental Model-based Policy Optimization (IMPO) algorithm for practical implementation.

Main Results:

• IMPO demonstrates superior performance compared to state-of-the-art model-based RL methods.
• The algorithm achieves significant improvements in sample efficiency across various control benchmarks.
• Experimental validation confirms the effectiveness of the incremental update scheme.

Conclusions:

• The proposed incremental update scheme enhances stability and performance in model-based RL.
• IMPO offers a practical and efficient solution for complex control problems.
• This work advances the reliability and sample efficiency of model-based RL approaches.