Near real-time online reinforcement learning with synchronous or asynchronous updates.

Area of Science:

• Control Systems Engineering
• Machine Learning
• Artificial Intelligence

Background:

• Reinforcement Learning (RL) is effective for controlling complex systems but faces limitations in practical applications due to challenges in interleaving learning and interaction.
• Integrating neural network complexity with real-time learning capabilities remains a significant hurdle for widespread RL adoption in control systems.

Purpose of the Study:

• To propose and validate an online Reinforcement Learning (RL) solution that addresses the limitations of interleaving environment interaction and learning steps.
• To enable near real-time learning capabilities for complex and unknown dynamical systems using RL.

Main Methods:

• An online learning solution is developed, encoding system states using past signals and reference model/input states.
• Value function and controller neural networks are trained online via backpropagation using system interaction data.
• The methodology is tested using a model-reference tracking control problem in both simulation and experimental hardware setups.

Main Results:

• The proposed online RL methodology is demonstrated to be valid through simulation and experimental case studies.
• Performance operation times were compared between synchronous and asynchronous updates using two high-level software packages.
• Analysis of software challenges and code runtime numbers provides insights into practical implementation.

Conclusions:

• Online synchronous RL shows strong potential for lower-order systems with fast dynamics, aligning with real-time requirements.
• Asynchronous online RL facilitates scaling to higher-dimensional systems and faster dynamics, even in non-hard real-time scenarios.