Model-Free Reinforcement Learning by Embedding an Auxiliary System for Optimal Control of Nonlinear Systems.

Area of Science:

• Control Systems Engineering
• Machine Learning
• Nonlinear Dynamics

Background:

• Optimal control for continuous-time nonlinear systems with unknown dynamics presents significant challenges.
• Externally added probing noise can induce oscillations, complicating the learning process in reinforcement learning algorithms.

Purpose of the Study:

• To propose a novel integral reinforcement learning (IRL) algorithm for optimal control of continuous-time nonlinear systems.
• To address the challenge of rejecting oscillations caused by probing noise during the learning phase.

Main Methods:

• An auxiliary trajectory is embedded as an exciting signal to decompose the state trajectory.
• A model-free policy iteration (PI) algorithm is developed using decoupled trajectories, alternating policy evaluation and improvement.
• An external input is introduced in the policy improvement step to bypass the need for input-to-state dynamics.

Main Results:

• The algorithm is implemented using an actor-critic structure with sequential updates of critic and actor neural network weights via least-squares methods.
• Convergence of the IRL algorithm and stability of the closed-loop system are mathematically guaranteed.
• Effectiveness is demonstrated through two illustrative examples.

Conclusions:

• The proposed IRL algorithm effectively solves the optimal control problem for continuous-time nonlinear systems with unknown dynamics.
• The novel approach successfully rejects oscillations from probing noise, ensuring stable learning and system performance.
• The method offers a robust solution for complex control challenges in dynamic systems.