An Advanced Optimal Tracking Control for Nonlinear Discrete-Time Systems Based on (N + 1)-Step Gradient Learning

In this article, to address the issue of accelerating convergence performance and eliminating the tracking error, an advanced optimal control method for nonlinear discrete-time systems is investigated based on an improved N-step [( ${N} +1$ )-step] gradient learning algorithm. Independent of the discount factor, this article introduces a novel tracking error index without quadratic input terms for the steady-state and convergence performances, which obtains the optimal control policy without calculating the reference control input. Compared with classic N-step gradient learning algorithms with infinite future reward assumption, the proposed algorithm investigates the (N +1)-step return with a fixed N and a step forward for finite tracking problems based on a long-term weighting parameter. Based on the above theory, value iteration (VI) and policy iteration (PI) methods are utilized to derive the convergence, monotonicity, optimality, and stability properties of the proposed algorithm, which can be conducted without the traditional assumption of zero initial functions. In the implementation of the algorithms, the actor-critic structure, constructed by four neural networks, is established to approximate the states, the value functions, and the control policy, respectively. Three simulation experiments on a helicopter system validate the efficacy and practicality of the control methods in addressing nonlinear optimal tracking challenges.