Storage capacity bounds in multilayer neural networks

Area of Science:

• Computational neuroscience
• Machine learning theory

Background:

• Neural networks are computational models inspired by biological nervous systems.
• Understanding the storage capacity of neural networks is crucial for designing efficient learning systems.

Purpose of the Study:

• To derive general analytic expressions for the lower and upper bounds of storage capacity in specific multilayer neural network architectures.
• To analyze special cases like committee and parity machines.
• To explain discrepancies between theoretical predictions and simulation results.

Main Methods:

• Derivation of general analytic expressions for storage capacity bounds.
• Analysis of specific network architectures with variable input-hidden weights and fixed output functions.
• Comparison of theoretical results with existing replica calculations and simulation data.

Main Results:

• Established lower and upper bounds for the storage capacity of the studied neural network models.
• Identified that simulations yield storage capacities slightly above the lower bound.
• Provided an explanation for the observed simulation results.

Conclusions:

• The derived analytic expressions offer a theoretical framework for understanding neural network storage capacity.
• Simulation results indicate potential for improvement in storage capacity beyond current observed levels.
• Further research can focus on implementing strategies to enhance network storage capacity based on these findings.