Spike-based Q-learning in a non-von Neumann architecture.

Area of Science:

• Computer Science
• Artificial Intelligence
• Neuroscience

Background:

• Von Neumann architectures face data-transfer bottlenecks and high power consumption due to memory-compute separation.
• Reinforcement learning (RL) workloads, especially Q-learning, require frequent updates across large state-action spaces, exacerbating these issues.
• Spiking neural networks (SNNs) offer computational efficiency through event-driven, sparse processing.

Purpose of the Study:

• To propose a hardware-feasible, non-von Neumann architecture based on SNNs for efficient Q-learning.
• To overcome the limitations of traditional architectures for RL tasks.
• To leverage SNNs' sparse processing for enhanced computational efficiency.

Main Methods:

• Developed a non-von Neumann architecture mapping states/actions to neurons and Q-values to synapses.
• Implemented a lateral inhibition structure for identifying maximum Q-values for updates.
• Incorporated a delay circuit for temporal consistency and local learning signals for targeted synapse updates.

Main Results:

• Simulations on the Cart-pole benchmark demonstrated stable learning performance.
• The architecture achieved comparable accuracy to software-based Q-learning with sufficient bit precision.
• Effective learning was observed even under low-bit precision conditions.

Conclusions:

• The proposed SNN-based non-von Neumann architecture effectively performs Q-learning.
• This approach significantly reduces data-transfer bottlenecks and power consumption for RL workloads.
• The architecture shows promise for efficient, hardware-accelerated reinforcement learning applications.