Neural Networks with a Redundant Representation: Detecting the Undetectable

Area of Science:

• Computational neuroscience
• Machine learning theory
• Statistical physics

Background:

• Sejnowski machines learn features through contrastive divergence.
• Dense associative memories store patterns using Hebbian learning.
• Associative memory capacity scales with neuron count and interaction order (N^{P-1}).

Purpose of the Study:

• To establish a dual representation between Sejnowski machine features and associative memory patterns.
• To investigate the pattern recognition capabilities of associative memories under low-load conditions.
• To explain the high performance of modern neural networks in pattern recognition.

Main Methods:

• Theoretical analysis of a three-layer Sejnowski machine.
• Mathematical modeling of dense associative memories with P-wise interactions.
• Replica symmetric approximation, signal-to-noise analysis, and Monte Carlo simulations.

Main Results:

• Learned features from contrastive divergence dual as patterns in a P=4 associative memory.
• Associative networks operating below saturation (linear pattern scaling) achieve pattern recognition below the signal-to-noise threshold.
• A P=4 network retrieves O(1) information amidst O(sqrt[N]) noise in the large N limit.

Conclusions:

• A redundant pattern representation in low-load associative memories enables robust information retrieval.
• This mechanism explains the impressive pattern recognition abilities of advanced neural networks.
• The findings bridge theoretical concepts in machine learning and neuroscience.