Entropy-Aware Model Initialization for Effective Exploration in Deep Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Effective exploration is crucial for deep reinforcement learning (DRL) agent performance.
• Insufficient exploration leads to poor data quality and policy degradation.
• Initial entropy significantly impacts exploration, especially during early learning stages.

Purpose of the Study:

• To investigate the effect of initial entropy on exploration in DRL.
• To address limitations in controlling initial entropy for discrete action spaces.
• To propose a novel strategy for enhancing exploration in DRL.

Main Methods:

• Analysis of initial entropy distributions across various tasks.
• Development of an entropy-aware model initialization strategy.
• Experimental evaluation of the proposed strategy on discrete action space tasks.

Main Results:

• Low initial entropy increases the likelihood of learning failures.
• Initial entropy distributions are often biased towards low values, inhibiting exploration.
• The proposed entropy-aware initialization significantly reduces learning failures.
• The strategy enhances DRL performance, stability, and learning speed.

Conclusions:

• Initial entropy is a critical, yet difficult to control, factor in DRL.
• Entropy-aware model initialization is an effective method to improve exploration.
• The proposed strategy offers a practical solution for enhancing DRL agent learning.