Actor-Critic Off-Policy Learning for Optimal Control of Multiple-Model Discrete-Time Systems

Area of Science:

• Control Engineering
• Machine Learning
• Artificial Intelligence

Background:

• Optimal tracking control is crucial for complex systems.
• Existing methods often require system dynamics knowledge.
• Human neurocognitive experiments inspire new control strategies.

Purpose of the Study:

• Develop a model-free, off-policy reinforcement learning algorithm.
• Address optimal tracking control for multiple-model linear discrete-time systems.
• Improve data efficiency and tuning speed in control systems.

Main Methods:

• Utilized an adaptive self-organizing map neural network to identify system behaviors.
• Implemented an off-policy iteration algorithm generalized for multiple-model learning.
• Employed partially weighted value functions to represent system states.
• Dynamically added new models upon detecting unobserved system behavior changes.

Main Results:

• The algorithm successfully performs optimal tracking control without prior system dynamics knowledge.
• Demonstrated increased data efficiency and faster tuning through experience reuse.
• Validated performance via two numerical examples.

Conclusions:

• The proposed model-free, off-policy reinforcement learning algorithm effectively solves optimal tracking control for multiple-model systems.
• The adaptive self-organizing map and generalized off-policy iteration enhance learning capabilities.
• This approach offers a promising direction for adaptive and efficient control systems.