The capacity of feedforward neural networks

Area of Science:

• Computational Neuroscience
• Machine Learning Theory
• Artificial Intelligence

Background:

• Estimating and comparing neural network architecture capabilities is a long-standing challenge.
• Existing methods lack quantitative measures for functional capacity.
• Understanding capacity is crucial for optimizing learning and data representation.

Purpose of the Study:

• To define and quantify the computational capacity of neural network architectures.
• To establish a theoretical framework for comparing different network designs.
• To investigate the relationship between architecture and the number of functions a network can compute.

Main Methods:

• Defined architecture capacity as the binary logarithm of computable functions.
• Analyzed layered, fully-connected architectures of linear threshold neurons.
• Developed new techniques including multiplexing, enrichment, and stacking.
• Derived a cubic polynomial formula for capacity based on layer sizes.

Main Results:

• Capacity is fundamentally a cubic polynomial of layer sizes: C(n1,…,nL)=∑k=1L-1min(n1,…,nk)nk*n(k+1).
• Smaller layers act as bottlenecks, significantly constraining capacity.
• Identified architectures with maximal or minimal capacity under various constraints.
• Demonstrated that shallow networks generally have higher capacity than deep networks.

Conclusions:

• The derived capacity formula provides a quantitative tool for neural network design.
• Introduced the concept of structural regularization for deep architectures.
• Deep networks, while having lower capacity, compute more regular and potentially more meaningful functions.
• The findings offer insights into optimizing network structure for specific computational tasks.