Off-Policy Interleaved Q -Learning: Optimal Control for Affine Nonlinear Discrete-Time Systems

Area of Science:

• Control Theory
• Machine Learning
• Nonlinear Systems

Background:

• Optimal control of affine nonlinear discrete-time (DT) systems presents significant challenges due to unknown dynamics and the need for off-policy learning.
• Existing on-policy Q-learning methods for these systems have limitations, including potential bias introduced by probing noises required for persistent excitation.

Purpose of the Study:

• To develop a novel off-policy interleaved Q-learning algorithm for optimal control of affine nonlinear DT systems.
• To rigorously prove the convergence of the proposed algorithm and demonstrate its ability to provide unbiased solutions.

Main Methods:

• A review and rigorous convergence proof of on-policy Q-learning for affine nonlinear DT systems.
• Analysis of solution bias in Q-function-based Bellman equations with probing noises.
• Introduction of a behavior control policy and development of an off-policy Q-learning algorithm.
• Implementation of three neural networks within an actor-critic framework using the interleaved Q-learning approach.

Main Results:

• The proposed off-policy interleaved Q-learning algorithm is derived and its convergence is rigorously proven.
• The algorithm effectively addresses challenges posed by unknown dynamics and off-policy learning in nonlinear DT systems.
• Simulation results confirm the effectiveness of the novel algorithm in solving optimal control problems.

Conclusions:

• The novel off-policy interleaved Q-learning algorithm offers a robust solution for optimal control of affine nonlinear discrete-time systems.
• The method successfully overcomes limitations of previous approaches, providing convergence guarantees and unbiased solutions.
• The findings are validated through simulations, demonstrating practical applicability.