Hierarchical Optimal Synchronization for Linear Systems via Reinforcement Learning: A Stackelberg-Nash Game...

Area of Science:

• Control Systems Engineering
• Game Theory
• Optimization

Background:

• Real-world systems often feature agents with sequential decision-making advantages.
• Hierarchical control structures with asymmetric agent roles present unique challenges in optimal synchronization.

Purpose of the Study:

• To formulate and solve a novel hierarchical optimal synchronization problem for linear systems from a Stackelberg-Nash game perspective.
• To address the complexities arising from asymmetric agent roles and coupled Hamilton-Jacobi-Bellman (HJB) equations.

Main Methods:

• Formulation of a Stackelberg-Nash game for a hierarchical linear system with one major and multiple minor agents.
• Establishment and analysis of coupled Hamilton-Jacobi-Bellman (HJB) equations to find optimal controllers.
• Development of a two-level value iteration (VI) reinforcement learning (RL) algorithm, utilizing neural networks (NNs) and gradient descent, to solve the HJB equations without requiring complete system matrices.

Main Results:

• The derived HJB equations' solutions are proven to be stable and constitute the Stackelberg-Nash equilibrium.
• The proposed two-level VI RL algorithm demonstrates convergence to optimal values.
• The RL algorithm effectively bypasses the need for complete system state information.

Conclusions:

• The developed Stackelberg-Nash game framework and RL-based solution effectively address hierarchical optimal synchronization in linear systems with asymmetric agents.
• The proposed two-level VI algorithm offers a practical and convergent method for solving complex, coupled HJB equations in control systems.
• The use of neural networks for value function approximation enables implementation without full system knowledge, verifying the approach's effectiveness through an illustrative example.