On Transforming Reinforcement Learning With Transformers: The Development Trajectory

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Transformers, initially for NLP, show promise in Computer Vision.
• Their powerful modeling capabilities are being explored for Reinforcement Learning (RL).
• Transformer-based RL (TRL) presents a new paradigm for intelligent agents.

Purpose of the Study:

• To survey and analyze recent advances in transformer-based RL (TRL).
• To categorize TRL developments into architecture enhancements and trajectory optimizations.
• To identify key applications and future research trends in TRL.

Main Methods:

• Dissecting recent literature on TRL.
• Categorizing TRL methods into architecture enhancements and trajectory optimizations.
• Examining TRL applications in robotics, text-based games, navigation, and autonomous driving.

Main Results:

• Architecture enhancements improve RL agent/environment modeling but face traditional RL limitations.
• Trajectory optimization methods leverage transformers for sequence modeling and policy extraction from datasets.
• TRL demonstrates potential across diverse applications, from manipulation to autonomous systems.

Conclusions:

• TRL offers significant advancements over traditional Deep RL techniques.
• Future research should address TRL limitations and explore novel applications.
• This survey provides a comprehensive overview and roadmap for TRL research.