Integral reinforcement learning for continuous-time input-affine nonlinear systems with simultaneous invariant...

Area of Science:

• Control Theory
• Machine Learning
• Nonlinear Systems

Background:

• Continuous-time nonlinear optimal control problems present significant challenges.
• Existing reinforcement learning (RL) methods often struggle with stability and exploration in these complex systems.

Purpose of the Study:

• To develop novel integral reinforcement learning (I-RL) algorithms for continuous-time (CT) nonlinear optimal control.
• To ensure stable exploration and convergence properties for model-free learning in input-affine systems.

Main Methods:

• Extension of exploration, integral temporal difference, and invariant admissibility concepts to CT nonlinear systems.
• Development of integral policy iteration (I-PI) and invariantly admissible PI (IA-PI) methods.
• Proposal of three online I-RL algorithms: explorized I-PI and integral Q-learning I, II.

Main Results:

• Demonstration of input-to-state stability (ISS) and invariant admissibility for closed-loop systems.
• Validation of proposed algorithms' convergence properties under excitation conditions.
• Verification of model-free capabilities and stable state-space exploration during online learning.

Conclusions:

• The proposed I-RL algorithms effectively solve CT nonlinear optimal control problems.
• Neural-network-based implementations are presented, demonstrating practical applicability.
• Design principles for safe exploration in reinforcement learning are established.