Maximizing theoretical and practical storage capacity in single-layer feedforward neural networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Artificial neural networks (ANNs) exhibit limitations in pattern storage and recall.
• Capacity constraints in ANNs stem from network size, architecture, pattern sparsity, and dissimilarity.
• Exceeding capacity leads to recall errors and catastrophic forgetting, a key challenge in continual learning.

Purpose of the Study:

• To theoretically characterize the maximum memory capacity of single-layer feedforward networks.
• To derive analytical expressions for this maximum capacity based on network parameters.
• To develop a pattern generation method that optimizes storage potential.

Main Methods:

• Derivation of analytical expressions for theoretical maximum memory capacity.
• Introduction of a grid-based construction and sub-sampling method for pattern generation.
• Validation of theoretical predictions through simulation results.

Main Results:

• Maximum theoretical memory capacity scales as (N/S)^S, where N is the number of units and S is pattern sparsity.
• Capacity is constrained by minimum pattern differentiability thresholds.
• A deterministic optimal pattern set construction systematically outperforms random generation.

Conclusions:

• The study provides a foundational framework for maximizing storage efficiency in neural networks.
• Findings support the development of data-efficient and sustainable artificial intelligence.
• Optimal pattern generation strategies can significantly enhance ANN memory capacity.