Optimal control under safety constraints and disturbances: a multi-step, off-policy adaptive dynamic programming...

Area of Science:

• Robotics and Control Systems
• Artificial Intelligence
• Optimization Theory

Background:

• Optimal control problems often involve complex dynamics and safety requirements.
• Traditional methods struggle with disturbances and ensuring constraint satisfaction.
• Accurate performance estimation is crucial but challenging under uncertainty.

Purpose of the Study:

• To introduce a robust multi-step, off-policy adaptive dynamic programming (ADP) approach for optimal control.
• To address challenges posed by external disturbances and critical safety constraints.
• To enhance the accuracy of performance function estimation and ensure a safety-performance trade-off.

Main Methods:

• Developed model-free and model-based variants of the off-policy adaptive dynamic programming.
• Employed interleaved training and prior models to mitigate underestimation in policy evaluation.
• Utilized dual critic neural networks to counteract terminal performance function underestimation.
• Transformed policy improvement into a constrained optimization task with a safety function.
• Designed an actor-critic-disturbance framework for handling safety constraints within a zero-sum game.

Main Results:

• The proposed method demonstrates effectiveness in solving optimal control problems with disturbances and safety constraints.
• Theoretical analysis confirms the convergence properties of the adaptive dynamic programming approach.
• Simulation results and practical experiments validate the method's performance and safety.

Conclusions:

• The multi-step, off-policy adaptive dynamic programming approach offers a viable solution for complex optimal control scenarios.
• The integration of dual critics and the actor-critic-disturbance framework enhances safety and performance.
• The method shows significant promise for real-world applications requiring safe and efficient control.