Can a Hebbian-like learning rule be avoiding the curse of dimensionality in sparse distributed data?

Area of Science:

• Computational Neuroscience
• Machine Learning

Background:

• The brain's sparse distributed representations are high-dimensional, posing challenges for traditional machine learning models due to the curse of dimensionality.
• Deep networks with many layers and backpropagation address this in data-rich scenarios, but the brain achieves similar feats with fewer layers.

Purpose of the Study:

• To investigate the hypothesis that Hebbian learning, specifically as implemented in Restricted Boltzmann Machines (RBMs), enables efficient processing of high-dimensional sparse data.
• To compare the classification performance of RBMs against backpropagation-trained neural networks on synthetic sparse datasets.

Main Methods:

• Generated several sparse datasets for testing.
• Trained Restricted Boltzmann Machines (RBMs) using their Hebbian-like learning rule, which focuses on correlations between non-zero values.
• Trained conventional neural networks using the backpropagation algorithm for comparison.

Main Results:

• Restricted Boltzmann Machines demonstrated robust generalization performance on the sparse datasets.
• Neural networks trained with backpropagation exhibited overfitting to the training data, indicating poorer generalization.
• RBMs' asymmetric learning rule, ignoring zero values, effectively bypasses the curse of dimensionality.

Conclusions:

• Hebbian learning, as utilized by RBMs, offers a viable mechanism for handling high-dimensional sparse data efficiently.
• RBMs present a promising alternative to traditional backpropagation networks for tasks involving sparse representations, particularly in biologically inspired AI.
• The findings support the idea that the brain may leverage Hebbian principles to overcome the curse of dimensionality.