Data-driven emergence of convolutional structure in neural networks

Area of Science:

• Machine Learning
• Computational Neuroscience
• Artificial Intelligence

Background:

• Exploiting data invariances is key for efficient learning in artificial and biological neural circuits.
• Understanding how neural networks discover representations that harness input symmetries is crucial for both machine learning and neuroscience.
• Convolutional neural networks (CNNs) leverage translation symmetry, driving early deep learning successes, but learning convolutions directly from translation-invariant data remains challenging.

Purpose of the Study:

• To investigate how initially fully connected neural networks can autonomously learn convolutional structures from translation-invariant data.
• To understand the role of input data's higher-order statistics in the emergence of learned convolutional patterns.
• To characterize the mechanism behind receptive field formation and its link to tensor decomposition.

Main Methods:

• Training initially fully connected neural networks on a discrimination task with translation-invariant data.
• Analyzing the emergent receptive field structures within the trained networks.
• Designing data models that capture the non-Gaussian, higher-order local structure characteristic of natural images.
• Employing analytical and numerical methods to characterize the pattern formation mechanism and its relation to tensor decomposition.

Main Results:

• Initially fully connected networks learned localized, space-tiling receptive fields, mimicking CNN filters.
• The emergence of these convolutional structures was triggered by the non-Gaussian, higher-order local structure of the input data.
• A link was established between receptive field formation and the tensor decomposition of higher-order input correlations.

Conclusions:

• Fully connected networks can learn convolutional representations directly from data exhibiting specific higher-order statistics.
• The non-Gaussian structure of natural images plays a critical role in the development of feature detectors.
• Findings offer insights into low-level feature detection in sensory systems and the impact of higher-order statistics on neural network learning.