Recurrent networks with soft-thresholding nonlinearities for lightweight coding.

Area of Science:

• Computational Neuroscience
• Information Theory
• Machine Learning

Background:

• A key hypothesis in neural processing posits that sensory networks efficiently encode input signals.
• Efficient coding aims to minimize redundancy and optimize information transmission.
• Understanding how networks adapt to input statistics is crucial for deciphering neural computation.

Purpose of the Study:

• To demonstrate how a nonlinear recurrent network can achieve optimal efficient coding of afferent inputs and their history.
• To develop a method for producing lightweight codes that satisfy both sparsity (L1) and energy (L2) constraints.
• To investigate the role of nonlinearities in enabling history-sensitive neural encoding.

Main Methods:

• Formulated the efficient coding problem as a non-smooth convex optimization problem within a linear coding paradigm.
• Employed a proximal gradient descent technique to derive the network's solution.
• Utilized gradient-based local learning for training network connection weights, assuming slower learning than recurrent dynamics.

Main Results:

• The optimal code is realized by a recurrent network featuring a nonlinear soft thresholding operator.
• Network training converges to optimal codes and weights through an alternative minimization procedure.
• Addition of thresholding nonlinearities enables the network to produce lightweight, history-sensitive encoding.

Conclusions:

• Nonlinear recurrent networks offer an effective framework for efficient neural information processing.
• Soft thresholding nonlinearities are key to achieving lightweight and history-sensitive coding.
• This approach provides a biologically plausible mechanism for adaptive sensory encoding.