Non-linear Feature Extraction by Redundancy Reduction in an Unsupervised Stochastic Neural Network

Area of Science:

• Computational Neuroscience
• Machine Learning
• Information Theory

Background:

• Unsupervised feature extraction is crucial for understanding complex environments.
• Existing methods often rely on linear assumptions, limiting their applicability to nonlinear systems.
• Barlow's principle of redundancy reduction offers a theoretical framework for efficient sensory coding.

Purpose of the Study:

• To develop a novel learning algorithm for stochastic recurrent networks.
• To implement unsupervised feature extraction based on information-theoretic principles.
• To extend linear principal component analysis to nonlinear cases using redundancy minimization.

Main Methods:

• Defining unsupervised feature extraction as minimizing output layer redundancy while maximizing input-output information transfer.
• Developing a learning rule incorporating Hebbian and anti-Hebbian terms, weighted by information transmission and redundancy.
• Simulating the algorithm on a retina model to observe receptive field formation.

Main Results:

• The learning algorithm successfully extracts statistically independent features, achieving a factorial representation.
• The derived learning rule provides an information-theoretic interpretation of Hebb's rule.
• Simulations demonstrated the formation of decorrelated receptive fields in a retina model, extending linear PCA to nonlinear scenarios.

Conclusions:

• The proposed algorithm effectively performs unsupervised feature extraction by minimizing redundancy.
• This work provides a direct implementation of Barlow's principle for nonlinear feature learning.
• The findings offer a novel approach to understanding receptive field formation and sensory coding.