Stimulus sampling as an exploration mechanism for fast reinforcement learning.

Area of Science:

• Computational Neuroscience
• Machine Learning

Background:

• Reinforcement learning (RL) in neural networks necessitates exploration mechanisms for novel states using nonspecific reward signals.
• Current RL models often employ intrinsic noise (synaptic or neuronal), which can hinder learning in population coding or mean firing rate representations.
• Balancing learning speed and reliability with intrinsic noise requires delicate parameter tuning.

Purpose of the Study:

• To investigate an alternative exploration mechanism for RL in neural networks that bypasses the need for intrinsic noise.
• To demonstrate that naturally occurring synaptic plasticity from stimulus sampling can drive effective exploration.
• To propose a framework for converting supervised learning rules into RL rules for networks.

Main Methods:

• Implementing reinforcement learning in neural networks using ongoing synaptic plasticity triggered by stimulus sampling.
• Combining stimulus sampling with a reward attenuation mechanism.
• Comparing the performance of the proposed method with intrinsic noise-based learning rules (node and weight perturbation).

Main Results:

• Synaptic plasticity from stimulus sampling provides sufficient fluctuations for successful learning without intrinsic noise.
• The proposed Hebbian-like learning rule, combined with reward attenuation, achieves performance comparable to primates in visuomotor association tasks.
• Learning rules based on intrinsic noise were significantly slower than the proposed method.
• The performance advantage of the stimulus sampling approach was maintained across more complex tasks and network architectures.

Conclusions:

• Intrinsic noise is not essential for exploration in reinforcement learning within neural networks.
• Stimulus sampling and reward attenuation offer a robust framework for enabling reinforcement learning.
• This approach facilitates the conversion of single-cell supervised learning rules into network-level reinforcement learning rules.
• The proposed method enhances learning efficiency and reliability, outperforming traditional intrinsic noise-based techniques.