A theory of capacity and sparse neural encoding

Area of Science:

• Computational neuroscience
• Machine learning theory
• Statistical physics

Background:

• Neural maps are crucial for information processing in biological and artificial systems.
• Understanding the storage capacity of neural networks is fundamental to their function.
• Sparse activity in neural networks is a common biological phenomenon with potential computational benefits.

Purpose of the Study:

• To investigate the storage capacity of sparse neural maps.
• To determine the impact of target layer sparsity on memory storage.
• To explore the feasibility of encoding and decoding memories using local learning rules.

Main Methods:

• Mathematical analysis of sparse neural maps.
• Proof of a phase transition related to storage capacity.
• Investigation of encoding and decoding using local learning rules (e.g., Hebb rule).
• Analysis based on properties of random polytopes and sub-gaussian random vectors.

Main Results:

• A phase transition in storage capacity (K) was mathematically proven.
• Sparsity in target layers paradoxically increases the storage capacity of the map.
• Memories can be reliably encoded and decoded using local learning rules.
• Results demonstrate robustness across various statistical assumptions.

Conclusions:

• Sparse neural maps exhibit enhanced storage capacity due to target layer sparsity.
• Local learning rules are sufficient for effective memory manipulation in these networks.
• The findings offer insights into biological neural computation and artificial intelligence.