Policy improvement by a model-free Dyna architecture.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Reinforcement learning (RL) policy improvement can be slow.
• Existing methods may require complex world models or extensive training.
• Efficient policy updates are crucial for real-world RL applications.

Purpose of the Study:

• To accelerate policy improvement in reinforcement learning.
• To develop a Dyna-style system integrating direct and indirect learning.
• To enhance the efficiency and robustness of RL agents.

Main Methods:

• A Dyna-style system combining temporal difference (TD) learning for direct updates and relative state values for indirect planning.
• Introduction of a simple average reward predictor within an actor-critic architecture for simulation (planning) mode.
• Utilizing relative state values (accumulated reward differences) to guide policy improvement.

Main Results:

• The proposed method demonstrated adaptability and robustness in controlling a pendulum system.
• In labyrinth exploration tasks, the model-free approach outperformed Dyna-Q and a connectionist adaptive heuristic critic.
• The system achieved superior performance in terms of reduced elapsed time and faster convergence rates.

Conclusions:

• The proposed reinforcement learning scheme effectively accelerates policy improvement.
• The integration of direct and indirect learning with a simplified planning model offers significant advantages.
• This approach shows promise for efficient and robust control in unknown dynamic environments.