An equivalence between adaptive dynamic programming with a critic and backpropagation through time.

Area of Science:

• Adaptive dynamic programming
• Reinforcement learning
• Control theory

Background:

• Dual Heuristic Programming (DHP) is an adaptive dynamic programming technique for learning critic functions using learned environment models.
• DHP is applied to optimize control problems in large, continuous state spaces.
• Standard DHP can exhibit divergence under specific conditions, particularly with greedy policies.

Purpose of the Study:

• To extend Dual Heuristic Programming (DHP) into a novel algorithm named Value-Gradient Learning (VGL(λ)).
• To establish theoretical convergence guarantees for the new VGL(λ) algorithm.
• To demonstrate the limitations of standard DHP and the advantages of the proposed VGL(λ) through experimental validation.

Main Methods:

• Extension of Dual Heuristic Programming (DHP) to create the Value-Gradient Learning (VGL(λ)) algorithm.
• Mathematical proof establishing the equivalence of a VGL(λ) instance to Backpropagation Through Time for Control (BPTT-C) with a greedy policy.
• Experimental evaluation including scenarios demonstrating DHP divergence and VGL(λ) convergence.

Main Results:

• Equivalence between VGL(λ) and BPTT-C with a greedy policy is proven.
• VGL(λ) demonstrates guaranteed convergence under smoothness conditions and a greedy policy when using a general smooth nonlinear function approximator for the critic.
• Experimental results highlight cases where standard DHP diverges, contrasting with the proven convergence of VGL(λ).

Conclusions:

• Value-Gradient Learning (VGL(λ)) provides a theoretically sound and convergent alternative to Dual Heuristic Programming (DHP) for control problems.
• The established link between VGL(λ) and BPTT-C offers new insights into adaptive dynamic programming techniques.
• VGL(λ) overcomes the convergence limitations of DHP, offering enhanced reliability for complex control tasks.