Continuous-Time Fitted Value Iteration for Robust Policies.

Area of Science:

• Control Theory
• Robotics
• Economics
• Differential Equations

Background:

• Solving the Hamilton-Jacobi-Bellman and Hamilton-Jacobi-Isaacs equations is crucial for optimal control and robust policies in continuous domains.
• These equations are fundamental in control, robotics, and economics, guiding decision-making for maximum reward and resilience against environmental adversaries.

Purpose of the Study:

• Introduce continuous fitted value iteration (cFVI) and robust fitted value iteration (rFVI) for continuous control problems.
• Develop algorithms that derive optimal policies and adversaries in closed form, simplifying differential equations without state or action discretization.

Main Methods:

• Leverage non-linear control-affine dynamics and separable rewards to derive analytic solutions.
• Employ value iteration for continuous states and actions, including adversarial scenarios.
• Apply cFVI and rFVI to Furuta pendulum and cartpole systems.

Main Results:

• Both cFVI and rFVI successfully obtain optimal policies for continuous control tasks.
• Sim2Real experiments confirm the policies' effectiveness on physical systems.
• Robust value iteration exhibited enhanced robustness against dynamic perturbations compared to deep reinforcement learning and non-robust methods.

Conclusions:

• cFVI and rFVI provide efficient, discretization-free solutions for continuous control and robust policy problems.
• The proposed robust value iteration method offers significant advantages in dynamic environments and perturbation resilience.
• The algorithms are validated on benchmark robotic systems, demonstrating practical applicability and superior robustness.