An Improved N-Step Value Gradient Learning Adaptive Dynamic Programming Algorithm for Online Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Dual heuristic programming (DHP) and value gradient learning (VGL) are effective for complex dynamics.
• VGL, inspired by temporal difference learning with eligibility traces (TD(λ)), excels in batch learning.
• Online learning with VGL is hindered by the need for an eligibility-trace-work-space matrix.

Purpose of the Study:

• Introduce N-step VGL (NSVGL), a dual-critic algorithm designed for efficient online learning.
• Eliminate the computational and memory burden of the eligibility-trace-work-space matrix.
• Enhance learning speed and performance in complex dynamic systems.

Main Methods:

• Developed NSVGL, a dual-critic extension of VGL, removing the need for the eligibility-trace-work-space matrix.
• Employed a dual-critic architecture where one critic adapts via TD(0) and the other via n-step TD(λ) gradients.
• Combined critic feedback to train an actor network for optimal decision-making.

Main Results:

• NSVGL facilitates online learning without the memory impediment of traditional VGL.
• The dual-critic structure enables faster learning and convergence compared to standard adaptive dynamic programming (ADP).
• Convergence proofs demonstrate monotonically nondecreasing gradients of value functions, converging to the optimum.

Conclusions:

• NSVGL offers a computationally efficient and faster alternative for online learning in complex dynamic environments.
• The dual-critic approach effectively combines immediate feedback with historical information for superior performance.
• Simulation studies confirm the superior performance of NSVGL over existing methods.