Hierarchical approximate policy iteration with binary-tree state space decomposition.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Approximate policy iteration (API) methods like LSPI struggle with large or continuous state spaces in reinforcement learning (RL).
• Existing API algorithms face challenges in achieving near-optimal policies for complex Markov decision processes (MDPs).

Purpose of the Study:

• To present a novel hierarchical approximate policy iteration (HAPI) method for RL in absorbing MDPs.
• To address the limitations of current API algorithms in handling large and continuous state spaces.
• To improve policy approximation accuracy and reduce computational complexity.

Main Methods:

• Developed a hierarchical API (HAPI) approach utilizing binary-tree state space decomposition.
• Employed a learning-based decomposition strategy for adaptive sample collection and state space partitioning.
• Applied API algorithms to sub-MDPs for approximating local optimal policies within the decomposed structure.

Main Results:

• The HAPI method successfully decomposed MDPs into a binary-tree structure of absorbing sub-MDPs.
• Local near-optimal policies were approximated with reduced complexity and enhanced precision.
• The combined global policy from HAPI demonstrated superior performance compared to single API algorithms.
• Evaluated HAPI on three learning control problems, including mobile robot path-tracking, showing improved results over LSPI and KLSPI.

Conclusions:

• HAPI offers a more effective approach for RL in absorbing MDPs with large state spaces.
• The hierarchical decomposition strategy enhances policy quality and computational efficiency.
• HAPI provides a significant advancement over traditional API methods for time-optimal learning control tasks.