LSTM-empowered reinforcement learning in Bi-level optimal control for nonlinear systems with uncertain dynamics

Area of Science:

• Robotics and Control Systems
• Artificial Intelligence
• Nonlinear System Dynamics

Background:

• Autonomous systems require robust control strategies for reliable operation in dynamic and uncertain environments.
• Traditional control methods often struggle with time-varying disturbances and complex nonlinear dynamics.
• Reinforcement learning (RL) and deep learning offer promising avenues for adaptive control but require careful integration and stability guarantees.

Purpose of the Study:

• To develop a novel bi-level optimization framework for optimal control of nonlinear continuous-time systems with uncertain dynamics.
• To integrate Long Short-Term Memory (LSTM) networks with an actor-critic reinforcement learning (RL) architecture for enhanced adaptive control.
• To achieve robust trajectory tracking without the need for offline training, ensuring adaptability to time-varying disturbances.

Main Methods:

• A bi-level optimization framework combining Hamiltonian-based optimal control with online uncertainty estimation.
• Utilizing an actor-critic RL architecture where the master level optimizes control policies (HJB-inspired) and the slave level uses LSTM networks for dynamic uncertainty estimation.
• Implementing rigorous stability analysis to prove uniform ultimate boundedness of the tracking error.

Main Results:

• Demonstrated superior tracking precision, energy efficiency, and disturbance rejection capabilities compared to conventional adaptive control and model-based virtual reference trajectory schemes.
• Validated the framework's effectiveness through extensive simulations on a skid-steering tracked robot executing diverse trajectories.
• Confirmed the framework's ability to handle time-varying disturbances and adapt to uncertain system dynamics.

Conclusions:

• The proposed bi-level optimization framework offers a computationally efficient and theoretically grounded solution for optimal control of nonlinear systems with uncertainties.
• This approach advances the paradigm of RL-based optimal control, providing a scalable solution for autonomous systems operating in unpredictable environments.
• The integration of LSTM networks and actor-critic RL ensures robust and adaptive trajectory tracking, enhancing system performance and reliability.