Recent Advances on Off-Policy Reinforcement Learning for Optimization Control

Area of Science:

• Artificial Intelligence
• Control Systems Engineering
• Machine Learning

Background:

• Reinforcement learning (RL) is a key AI technique for optimization and control.
• Two main RL frameworks exist: on-policy and off-policy (OffP-RL).
• OffP-RL addresses exploration limitations in on-policy methods, enhancing practicality.

Purpose of the Study:

• To review recent advancements in off-policy reinforcement learning for control.
• To classify OffP-RL control methods based on the number of players/controllers.
• To analyze applications and future directions of OffP-RL control.

Main Methods:

• Classification of OffP-RL control methods into single-player, two-player, and multiplayer categories.
• Review of recent literature on each category.
• Analysis of system data generation in relation to behavior and target policies.

Main Results:

• Single-player OffP-RL focuses on learning optimal control policies to minimize performance indices.
• Two-player OffP-RL commonly addresses H-infinity control and zero-sum games, seeking Nash equilibria.
• Multiplayer OffP-RL encompasses single systems with multiple inputs and multiagent systems with independent inputs.

Conclusions:

• Off-policy RL provides a more practical approach to control problems compared to on-policy methods.
• The classification into single-, two-, and multiplayer scenarios offers a structured overview of the field.
• Further research into OffP-RL applications and future work is warranted.