A Hybrid Online Off-Policy Reinforcement Learning Agent Framework Supported by Transformers

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Reinforcement learning (RL) enables agents to learn optimal policies via environmental interaction.
• Traditional RL faces challenges like extensive data requirements and long-term credit assignment.
• Transformers have shown promise in addressing RL limitations, particularly in offline settings.

Purpose of the Study:

• To propose a framework enhancing online off-policy reinforcement learning agents using Transformers.
• To address data inefficiency and credit assignment problems in RL through self-attention mechanisms.
• To improve the training efficiency of Transformer-based RL agents in data-scarce or unknown environments.

Main Methods:

• A hybrid agent combining an online off-policy RL agent and an offline Transformer agent (Decision Transformer architecture).
• Sequential exchange of the experience replay buffer between the online and offline agents.
• Utilizing self-attention mechanisms inherent in the Transformer architecture.

Main Results:

• Improved learning training efficiency in the initial iterations for the hybrid agent.
• Enhanced training for Transformer-based RL agents, particularly in scenarios with limited data.
• Demonstrated effectiveness of the mixed policy approach in addressing RL challenges.

Conclusions:

• The proposed framework effectively enhances online off-policy RL training using Transformers.
• The hybrid agent architecture and experience replay exchange improve learning efficiency and data utilization.
• This approach offers a promising solution for RL in challenging environments with limited data or unknown dynamics.