Efficient model learning methods for actor-critic control.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) algorithms, particularly actor-critic methods, are essential for sequential decision-making.
• Approximating value and policy functions efficiently is a key challenge in RL.
• Model-based RL approaches can improve sample efficiency but often require accurate models.

Purpose of the Study:

• To develop novel actor-critic algorithms that enhance learning efficiency.
• To integrate local linear regression (LLR) with learned process models for improved policy updates.
• To evaluate the performance of these new algorithms against standard methods.

Main Methods:

• Two new actor-critic algorithms are proposed, both employing local linear regression (LLR).
• Both algorithms learn an internal process model for more efficient updates.
• The first algorithm features a novel model-based update for actor parameters; the second learns a reference model for control action calculation.

Main Results:

• The proposed algorithms demonstrated faster learning compared to a standard actor-critic algorithm.
• The integration of LLR with learned process models proved effective for policy updates.
• Performance was validated on the challenging pendulum swing-up problem.

Conclusions:

• The novel actor-critic methods offer a more efficient approach to reinforcement learning.
• Learning a process model alongside function approximation significantly accelerates policy learning.
• These findings suggest a promising direction for improving model-based reinforcement learning techniques.