Adaptive exploration strategy in reinforcement learning based on Q-values and environmental cognition.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Reinforcement learning (RL) excels in sequential decision-making but struggles with exploration-exploitation balance.
• Existing methods often use single signals, leading to inefficient exploration and suboptimal solutions.

Purpose of the Study:

• To develop an adaptive exploration strategy for reinforcement learning.
• To improve learning efficiency and performance by addressing limitations of current exploration techniques.

Main Methods:

• Propose Var, an adaptive exploration strategy using Q-values and environmental cognition (Q-value difference and novelty bonus).
• Integrate Var into Q-learning (Var-QL) and Deep Q-Networks (Var-DQN).
• Introduce LRU-Var-QL for large tabular discrete environments, incorporating a Least Recently Used (LRU) cache.

Main Results:

• Var encourages broad and deep exploration initially, transitioning to exploitation.
• LRU-Var-QL demonstrated reduced catastrophic actions on FrozenLake-v1.
• Var-QL and Var-DQN achieved higher cumulative returns and faster learning on Atari games compared to baselines.

Conclusions:

• The proposed Var strategy enhances reinforcement learning exploration.
• Var-based methods show significant improvements in learning speed and performance across diverse environments.