Constructing Temporally Extended Actions through Incremental Community Detection.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Hierarchical reinforcement learning (HRL) utilizes temporally extended actions (options) to improve learning efficiency.
• Automatic formation and adaptability of options during learning remain significant challenges in HRL.
• Existing methods for option construction often lack dynamic adaptation to the learning environment.

Purpose of the Study:

• To present a novel algorithm for online creation and enhancement of options in HRL.
• To improve the adaptability and quality of options during the learning process.
• To enhance overall reinforcement learning performance through improved option strategies.

Main Methods:

• The algorithm operates on detected communities within the learning environment's state transition graph.
• Initial options are constructed from sample data to establish a baseline for online learning.
• A rule-based community revision algorithm dynamically updates graph partitions to tune existing options.

Main Results:

• Experimental results demonstrate that initially formed options can underperform in complex environments.
• The proposed online adaptation strategy significantly improves option quality and learning outcomes.
• The adaptive HRL approach outperforms standard flat reinforcement learning methods in tested problems.

Conclusions:

• The presented algorithm effectively addresses the challenge of option adaptability in HRL.
• Online refinement of options based on environment structure leads to superior learning performance.
• This work offers a promising direction for developing more robust and efficient HRL systems.