Sampling Efficient Deep Reinforcement Learning Through Preference-Guided Stochastic Exploration.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Stochastic exploration is crucial for deep Q-network (DQN) success.
• Existing exploration methods often introduce bias by heuristically selecting actions or coupling sampling with action values.

Purpose of the Study:

• To propose a novel preference-guided epsilon-greedy exploration algorithm for DQN.
• To facilitate efficient exploration in DQN without introducing additional bias.

Main Methods:

• A dual architecture with two branches: a standard DQN branch and a preference branch.
• The preference branch learns the action preferences implicitly followed by the DQN.
• Theoretical proof that the policy improvement theorem holds for the preference-guided epsilon-greedy policy.

Main Results:

• Experimental validation showing the inferred action preference distribution aligns with value landscapes.
• The preference-guided exploration encourages diverse action selection, sampling high-value actions more frequently while still exploring lower-value actions.
• Comprehensive benchmarking against DQN variants across nine environments.

Conclusions:

• The proposed preference-guided epsilon-greedy exploration method significantly enhances DQN performance.
• The approach improves convergence speed compared to existing DQN variants.
• This method offers a bias-free and effective strategy for exploration in deep reinforcement learning.