Online Model-Free n-Step HDP With Stability Analysis.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Temporal-difference (TD) learning, particularly TD(λ), is powerful for optimal control but often requires offline training with eligibility traces.
• Existing methods like heuristic dynamic programming (HDP) and action-dependent HDP (ADHDP) have limitations in terms of computational cost and training efficiency.

Purpose of the Study:

• To present a novel n-step adaptive dynamic programming (ADP) architecture, NSHDP(λ), combining TD(λ) with regular TD learning.
• To develop an online, model-free approach that reduces iterations and computational overhead compared to traditional methods.

Main Methods:

• Introduced NSHDP(λ), an architecture featuring three neural networks: a critic network (CN) with TD(0), a CN with n-step TD(λ) learning, and an actor network (AN).
• Employed a forward view learning approach, enabling online updates at each time step without eligibility traces.
• Utilized Lyapunov analysis to prove the stability and uniformly ultimately bounded (UUB) property of NSHDP(λ).

Main Results:

• NSHDP(λ) demonstrated reduced iterations and computational costs due to its online, model-free nature and lack of eligibility traces.
• The method showed stable performance, proven by Lyapunov analysis.
• Simulations on a complex nonlinear system, a 2-D maze, and an inverted pendulum benchmark confirmed NSHDP(λ)'s effectiveness compared to HDP and ADHDP(λ).

Conclusions:

• NSHDP(λ) offers an efficient and memory-light solution for optimal control problems, especially in model-free scenarios.
• The online, forward-view learning approach significantly enhances computational efficiency and reduces training requirements.
• The proven stability and strong performance across various benchmarks validate NSHDP(λ) as a promising ADP architecture.