Neural networks with local receptive fields and superlinear VC dimension

Area of Science:

• Computational Neuroscience
• Machine Learning Theory
• Artificial Neural Networks

Background:

• Local receptive field (LRF) neurons, such as radial basis function (RBF) and center-surround units, are fundamental components in feedforward neural networks.
• Understanding the Vapnik-Chervonenkis (VC) dimension is crucial for characterizing the generalization capacity and complexity of neural network models.
• Previous research has explored the VC dimension of networks with sigmoidal neurons, but less is known about networks employing LRF units.

Purpose of the Study:

• To investigate and establish lower bounds for the Vapnik-Chervonenkis (VC) dimension of feedforward neural networks with one hidden layer of local receptive field (LRF) neurons.
• To compare the VC dimension of LRF networks with those of sigmoidal networks and to address an open question regarding RBF neural networks.
• To derive new theoretical insights into the expressive power and generalization capabilities of neural networks utilizing physiologically relevant and computationally popular neuron models.

Main Methods:

• Analysis of feedforward neural networks with a single hidden layer composed of various types of local receptive field neurons.
• Derivation of Vapnik-Chervonenkis (VC) dimension bounds for networks with discrete center-surround receptive field neurons, difference of Gaussians neurons, and standard radial basis function (RBF) neurons.
• Establishing a superlinear lower bound of Omega(W log k) for the VC dimension, where W is the number of parameters and k is the number of hidden nodes.

Main Results:

• Demonstrated that the Vapnik-Chervonenkis (VC) dimension for feedforward neural networks with one hidden layer of local receptive field (LRF) neurons is superlinear.
• Established a lower bound of Omega(W log k) for the VC dimension of networks employing discrete center-surround, difference of Gaussians, and radial basis function (RBF) neurons.
• Showed that these derived bounds are larger than those known for similar architectures using sigmoidal neurons, and contrasted with newly derived linear upper bounds for single LRF neurons.

Conclusions:

• The Vapnik-Chervonenkis (VC) dimension of feedforward neural networks with common local receptive field (LRF) neurons, including RBF and center-surround types, grows superlinearly with network size.
• The established Omega(W log k) lower bound provides a significant theoretical result, particularly for RBF networks, and offers a more refined understanding of their capacity.
• These findings suggest that LRF-based neural networks may possess greater expressive power compared to sigmoidal networks of similar size, with implications for model selection and generalization analysis.