Reinforcement learning solution for HJB equation arising in constrained optimal control problem.

Area of Science:

• Control Theory
• Machine Learning
• Applied Mathematics

Background:

• Constrained optimal control problems necessitate solving the complex Hamilton-Jacobi-Bellman equation (HJBE).
• Traditional methods often face challenges with data efficiency and exploration.

Purpose of the Study:

• To propose a novel data-based, off-policy reinforcement learning (RL) method for solving HJBE and deriving optimal control policies.
• To address the insufficient exploration problem inherent in RL through off-policy learning.

Main Methods:

• Utilizing an off-policy reinforcement learning (RL) framework to learn from real system data.
• Employing an actor-critic neural network architecture with linearly independent basis functions for function approximation.
• Proving convergence by establishing equivalence to the successive approximation approach.

Main Results:

• Demonstrated the convergence of the proposed off-policy RL method.
• Proved the convergence of the implementation procedure incorporating function approximation.
• Verified the effectiveness of the method through computer simulations.

Conclusions:

• The proposed data-based off-policy RL method effectively learns HJBE solutions and optimal control policies from real system data.
• The method successfully addresses exploration limitations and shows convergence guarantees.
• Simulation results confirm the practical applicability and effectiveness of the approach.