QC_SANE: Robust Control in DRL Using Quantile Critic With Spiking Actor and Normalized Ensemble

Area of Science:

• Artificial Intelligence
• Robotics
• Computational Neuroscience

Background:

• Deep reinforcement learning (DRL) has advanced fields like online gaming and robotics.
• Continuous control problems present unique challenges for DRL agents.
• Existing methods like population coded spiking actor networks (PopSAN) have limitations.

Purpose of the Study:

• To introduce a novel DRL approach, Quantile Critic with Spiking Actor and Normalized Ensemble (QC_SANE), for continuous control problems.
• To leverage quantile loss for critic training and spiking neural networks for actor ensembles.
• To enhance robustness and performance in complex robotic tasks.

Main Methods:

• Developed QC_SANE, integrating quantile loss for critic and a spiking neural network (NN) for actor ensembles.
• Utilized scaled exponential linear unit (SELU) activation within the NN for internal normalization and robustness.
• Conducted empirical studies on MuJoCo-based environments featuring multijoint dynamics with contact.

Main Results:

• QC_SANE demonstrated superior training and testing performance compared to the state-of-the-art PopSAN.
• The proposed method achieved improved results in complex, contact-rich robotic simulations.
• Internal normalization via SELU contributed to the robustness of the spiking NN actors.

Conclusions:

• QC_SANE offers a significant advancement for deep reinforcement learning in continuous control.
• The integration of quantile critics and spiking actor ensembles enhances performance and robustness.
• This approach shows promise for tackling complex robotic manipulation and control tasks.