EASpace: Enhanced Action Space for Policy Transfer.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Traditional methods struggle with long-horizon reinforcement learning tasks due to inefficient exploration and credit assignment.
• Option-based multipolicy transfer methods exhibit limitations in exploring macro action duration and exploiting long-duration actions.

Purpose of the Study:

• To propose a novel algorithm, Enhanced Action Space (EASpace), for accelerating reinforcement learning by effectively utilizing suboptimal expert policies.
• To address the challenges of inefficient exploration and insufficient exploitation in long-horizon tasks through a new macro action formulation.

Main Methods:

• EASpace formulates each expert policy into multiple macro actions with varying execution times.
• Macro actions are directly integrated into the primitive action space.
• An intrinsic reward proportional to macro action execution time is introduced to incentivize exploitation.
• A learning rule similar to intra-option Q-learning is employed for improved data efficiency.

Main Results:

• Theoretical analysis confirms the convergence of the proposed learning rule.
• EASpace demonstrated efficiency in a grid-based game and a multiagent pursuit problem.
• The algorithm's effectiveness was validated through implementation in physical systems.

Conclusions:

• EASpace offers an effective approach to accelerate reinforcement learning by structuring exploration and improving credit assignment.
• The algorithm enhances the exploitation of useful long-duration macro actions, overcoming limitations of traditional methods.
• EASpace shows promise for real-world applications in complex control tasks and robotics.