Toward Generalization of Automated Temporal Abstraction to Partially Observable Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Temporal abstraction in reinforcement learning (RL) aims to accelerate learning by identifying recurring sub-policy patterns.
• Existing methods often assume fully observable environments, limiting their application to partially observable scenarios.
• The curse of dimensionality poses a significant challenge for automatic abstraction extraction in RL.

Purpose of the Study:

• To adapt existing automatic abstraction methods for partially observable reinforcement learning settings.
• To introduce novel belief state discretization techniques compatible with the enhanced abstraction mechanism.
• To improve the efficiency and applicability of temporal abstraction in complex, real-world RL problems.

Main Methods:

• Adaptation of the extended sequence tree method for model-based partially observable RL.
• Development and integration of belief state discretization strategies.
• Empirical evaluation on established benchmark problems to validate the approach.

Main Results:

• The modified extended sequence tree effectively handles a specific family of model-based partially observable RL problems.
• The proposed belief state discretization methods enhance the performance of the abstraction mechanism.
• Empirical results demonstrate the practical effectiveness of the developed abstraction method on benchmark tasks.

Conclusions:

• The adapted temporal abstraction method extends the applicability of automatic abstraction to partially observable RL.
• Belief state discretization is a crucial component for successful abstraction in these settings.
• The findings pave the way for more efficient RL agents in complex, uncertain environments.