Expressive probabilistic sampling in recurrent neural networks.

Area of Science:

• Computational Neuroscience
• Machine Learning
• Dynamical Systems

Background:

• Bayesian models of brain function propose neural activity represents samples from probability distributions for computation.
• A gap exists in understanding how neural dynamics can mechanistically sample from arbitrary distributions.

Approach:

• Utilized functional analysis and stochastic differential equations to determine minimal architectural needs for recurrent neural circuits to sample distributions.
• Investigated traditional sampler-only networks, identifying limitations in synaptic current and firing-rate dynamics for complex distributions.
• Proposed reservoir-sampler networks (RSNs) with separate output units for enhanced sampling capacity.

Key Points:

• Traditional neural sampling models have limited capacity for complex probability distributions.
• Reservoir-sampler networks (RSNs) demonstrate the ability to sample from arbitrary distributions.
• An efficient training method using denoising score matching enables RSNs to implement Langevin sampling.

Conclusions:

• RSNs provide a viable neural mechanism for sampling complex distributions.
• The proposed model advances the development of next-generation sampling-based brain models.
• Empirical validation shows RSNs can sample from diverse complex data distributions.