Unsupervised Learning of Temporal Abstractions With Slot-Based Transformers.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Reusable subroutines simplify complex decision-making and planning in reinforcement learning.
• Existing unsupervised methods for learning temporal abstractions process trajectories sequentially, limiting revision of subroutine boundaries with new information.

Purpose of the Study:

• To develop a novel, parallel approach for unsupervised discovery of temporal abstractions (subroutines) in reinforcement learning.
• To overcome the limitations of sequential processing in prior methods for subroutine discovery.

Main Methods:

• Proposed the slot-based transformer for temporal abstraction (SloTTAr), integrating sequence processing transformers with a slot attention module.
• Employed adaptive computation for learning the number of subroutines based on their empirical distribution.
• Developed a fully parallel approach for processing trajectories.

Main Results:

• SloTTAr outperforms strong baselines in discovering subroutine boundary points.
• The method effectively handles sequences with variable numbers of subroutines.
• Achieved up to a seven-fold increase in training speed compared to existing benchmarks.

Conclusions:

• SloTTAr offers a more efficient and effective solution for unsupervised subroutine discovery in reinforcement learning.
• The parallel and adaptive nature of SloTTAr enables faster learning and improved performance in complex planning tasks.