Dynamic sparse coding-based value estimation network for deep reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Systems

Background:

• Deep Reinforcement Learning (DRL) is crucial for control automation.
• Value Estimation Networks (VENs) in DRL are susceptible to catastrophic interference.
• Accurate value estimation is key to DRL performance.

Purpose of the Study:

• To propose a Dynamic Sparse Coding (DSC)-based VEN model.
• To improve value prediction accuracy and training efficiency in DRL.
• To address interference and redundant parameters in VENs.

Main Methods:

• Implemented DSC for sparse representations and dynamic gradients in VENs.
• Utilized dynamic thresholds for efficient weight pruning.
• Applied the DSC-VEN model to both discrete-action (Q-learning) and continuous-action (actor-critic) DRL.

Main Results:

• Achieved higher control performances compared to benchmark DRL algorithms.
• Demonstrated significant performance increases (e.g., >25% in Puddle World, ~10% in Hopper).
• Showcased efficient convergence with fewer episodes across different environments.

Conclusions:

• DSC-based VENs offer precise sparse representations for accurate value prediction.
• DSC effectively alleviates interference and prunes redundant parameters.
• The proposed method enhances DRL performance and training efficiency in diverse control tasks.