Accelerated Reinforcement Learning With Verifiable Excitation for Cubic Convergence

Area of Science:

• Control Systems Engineering
• Machine Learning
• Optimization Theory

Background:

• Reinforcement learning (RL) algorithms for linear systems often rely on persistent excitation (PE), limiting practical applications.
• Existing policy iteration (PI)-based RL methods exhibit quadratic convergence rates.

Purpose of the Study:

• To develop an accelerated reinforcement learning (RL) algorithm for discrete-time linear systems with unknown dynamics.
• To achieve cubic convergence without requiring persistent excitation (PE).
• To enable data-driven implementation through a verifiable excitation condition.

Main Methods:

• Proposed an accelerated reinforcement learning (RL) algorithm utilizing a midpoint-centered Lyapunov equation for cubic convergence.
• Introduced a verifiable excitation condition based on filter outputs for data-driven implementation without PE.
• Developed a data-driven procedure with a bisection rule to compute an initial admissible control gain.

Main Results:

• The accelerated RL algorithm demonstrates cubic convergence, surpassing existing quadratic rates.
• The proposed verifiable excitation condition ensures data informativeness for solving least-squares equations.
• Theoretical analysis confirms the cubic convergence and the effectiveness of the excitation condition.

Conclusions:

• The accelerated RL algorithm offers a significant improvement in convergence speed for discrete-time linear systems.
• The novel excitation condition facilitates practical, data-driven RL applications without PE.
• The method is validated through simulations, showing superior performance compared to existing approaches.