A Novel Topology Adaptation Strategy for Dynamic Sparse Training in Deep Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep reinforcement learning (DRL) is powerful but computationally intensive.
• Dynamic sparse training (DST) reduces demands but often degrades policy performance.
• Current DST methods prune based on connection magnitude, limiting effectiveness.

Purpose of the Study:

• To enhance policy performance in DRL while maintaining sparsity levels.
• To introduce a generic method for integrating into existing DST approaches.
• To address limitations of magnitude-based pruning and random connection generation in DST.

Main Methods:

• Developed a novel method for calculating connection importance in DRL models.
• Dynamically adjusted sparse network topology by dropping and introducing connections based on importance.
• Integrated the proposed method into two state-of-the-art DST approaches.

Main Results:

• Improved performance of two SOTA DST methods by up to 70% in episode return.
• Demonstrated enhanced average return across all episodes under various sparsity levels.
• Validated the method's effectiveness on eight widely used simulation tasks.

Conclusions:

• The proposed generic method effectively enhances DRL policy performance within DST frameworks.
• The importance-based connection adjustment preserves sparsity while boosting efficiency.
• This approach offers a significant advancement for practical DRL applications.