Sparsely Connected, Hebbian Networks with Strikingly Large Storage Capacities

Area of Science:

• Computational Neuroscience
• Artificial Neural Networks
• Memory Modeling

Background:

• Fully-connected networks (F-nets) face challenges in realistic biological memory modeling due to high synaptic densities and low storage capacity with simple learning rules.
• Existing low-connectivity auto-associative networks often rely on random projections, limiting their efficiency.
• Projective networks (P-nets) offer an alternative with nonrandom, multilayer structures and extremely low connectivities.

Purpose of the Study:

• To derive a lower bound on the storage capacities of a specific class of two-layer projective networks (P-nets).
• To evaluate the performance of P-nets with binary Hebbian synapses for biological memory modeling.
• To demonstrate the efficiency of P-nets in terms of connectivity and storage capacity compared to F-nets.

Main Methods:

• Development of a theoretical framework to derive storage capacity bounds for two-layer P-nets.
• Utilizing binary Hebbian synapses within the P-net architecture.
• Analysis of network performance with a 1% tolerance for spurious neurons.

Main Results:

• A P-net with 1000 synapses per neuron (approximately 2 x 10^6 neurons) can store over 1.5 x 10^6 training vectors.
• Each training vector consists of 20 active neurons, representing an information density of 0.25 bits per synapse.
• P-nets achieve significantly higher storage capacities at substantially lower connectivities compared to F-nets.

Conclusions:

• Projective networks (P-nets) present a viable and efficient architecture for constructing realistic partial models of biological memory.
• The derived theoretical bounds confirm the high storage capacity and low connectivity advantages of P-nets.
• P-nets with binary Hebbian synapses offer a promising direction for future research in neural network design for memory applications.