A Unified Experience Replay Framework for Spiking Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Energy-Efficient Computing

Background:

• Deep Reinforcement Learning (DRL) offers significant capabilities but suffers from high energy consumption.
• Spiking Neural Networks (SNNs) reduce energy use in DRL but face challenges with sample quality and performance in short simulations.
• Existing Spiking DRL methods struggle with the energy-performance tradeoff due to fixed replay buffers.

Purpose of the Study:

• To develop a generic resilient experience replay method for Spiking DRL.
• To address the tradeoff between energy consumption and model performance in Spiking DRL.
• To enhance sample quality and model performance without compromising energy efficiency.

Main Methods:

• Implemented a dynamic replay buffer that expands with training samples.
• Introduced adaptive buffer management to shrink the buffer and remove redundant samples.
• Integrated the method seamlessly into existing state-of-the-art Spiking DRL algorithms.

Main Results:

• Significantly enhanced the performance (return) of five SOTA Spiking DRL methods.
• Demonstrated improvements across sixteen diverse tasks and various simulation durations.
• Maintained energy efficiency without performance compromise.

Conclusions:

• The resilient experience replay method effectively resolves the energy-performance tradeoff in Spiking DRL.
• Dynamic and adaptive buffer management is crucial for improving sample efficiency in SNN-based RL.
• This approach offers a practical solution for deploying energy-efficient AI in real-world applications.