Transformer Decoder-Based Enhanced Exploration Method to Alleviate Initial Exploration Problems in Reinforcement...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Epsilon-greedy strategy is a common exploration technique in reinforcement learning.
• This strategy often results in extensive initial exploration and prolonged learning periods.
• Current methods to reduce exploration, like using expert data, have limitations in reducing the initial exploration range.

Purpose of the Study:

• To propose a novel method for reducing the initial exploration range in reinforcement learning.
• To enhance learning efficiency and agent performance by guiding early actions.
• To improve upon existing exploration techniques in reinforcement learning.

Main Methods:

• Pretraining a transformer decoder with extensive expert data.
• Using the pretrained model to guide agent actions during initial learning stages.
• Transitioning to the epsilon-greedy strategy after a learning threshold is met.

Main Results:

• The proposed method demonstrated approximately 2.5 times the average reward in the FreeStyle1 basketball game.
• Achieved a 26% higher win rate compared to the traditional Deep Q-Network (DQN) with epsilon-greedy strategy.
• Effectively reduced the initial exploration range and optimized learning times.

Conclusions:

• The pretrained transformer decoder method significantly enhances reinforcement learning performance.
• This approach offers a substantial improvement over traditional exploration techniques.
• The method effectively balances exploration and exploitation for faster, more efficient learning.