Prioritizing Useful Experience Replay for Heuristic Dynamic Programming-Based Learning Systems.

Area of Science:

• Robotics and Control Systems
• Artificial Intelligence
• Machine Learning

Background:

• Traditional adaptive dynamic programming controllers require extensive training due to low data efficiency.
• Prioritized experience replay (ER) enhances learning by focusing on crucial data samples.

Purpose of the Study:

• To integrate prioritized ER into heuristic dynamic programming (HDP) for improved learning efficiency.
• To develop a systematic approach for incorporating ER into HDP's critic and action networks.

Main Methods:

• Designed an ER tuple using one-time-step backward state-action pairs, eliminating the need for a model network.
• Systematically integrated prioritized ER into both critic and action networks of the HDP controller.
• Validated the approach on cart-pole and triple-link pendulum balancing tasks with controlled initial conditions.

Main Results:

• The proposed ER-integrated HDP approach improved success rates by 60.56% for cart-pole and 56.89% for triple-link pendulum balancing compared to traditional HDP.
• Demonstrated superior performance against traditional HDP and ER-based HDP methods.
• Provided theoretical convergence analysis to ensure control design stability.

Conclusions:

• Integrating prioritized ER into HDP significantly enhances learning efficiency and reduces training time for complex control tasks.
• The proposed method offers a stable and effective solution for adaptive dynamic programming controllers.
• This approach shows promise for real-world applications requiring efficient and robust control systems.