Efficient Actor-Critic Algorithm with Hierarchical Model Learning and Planning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Reinforcement Learning (RL) algorithms often face challenges with slow convergence and low sample efficiency.
• Hierarchical approaches can improve RL performance by structuring learning and planning processes.

Purpose of the Study:

• To propose and evaluate two novel efficient learning methods, AC-HMLP and RAC-HMLP, for enhancing convergence rate and sample efficiency in RL.
• To leverage hierarchical models for improved information utilization in RL algorithms.

Main Methods:

• Developed AC-HMLP and RAC-HMLP by integrating actor-critic algorithms with hierarchical model learning and planning.
• Employed local linear regression (LLR) for local models and linear function approximation (LFA) for global models within a hierarchical structure.
• Utilized both local and global models for sample generation during planning, with conditional application of the local model based on state-prediction error.

Main Results:

• AC-HMLP and RAC-HMLP demonstrated superior performance compared to three representative RL algorithms on benchmark problems.
• The proposed methods achieved significant improvements in both convergence rate and sample efficiency.
• The integration of local and global models effectively utilized information for accelerated learning.

Conclusions:

• AC-HMLP and RAC-HMLP represent effective advancements in RL, offering improved convergence and sample efficiency.
• Hierarchical model learning and planning, combined with actor-critic methods, provide a robust framework for complex RL tasks.