Deep neural networks have an inbuilt Occam's razor

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computational Neuroscience

Background:

• Overparameterized deep neural networks (DNNs) exhibit remarkable performance.
• The reasons for this success are attributed to the interplay between network architecture, training algorithms, and data structure.
• Disentangling these components is crucial for understanding supervised learning.

Purpose of the Study:

• To investigate the interplay between network architecture, training algorithms, and data structure in deep neural networks (DNNs).
• To apply a Bayesian framework to analyze the functions expressed by DNNs.
• To understand the inductive bias that contributes to the success of DNNs in supervised learning.

Main Methods:

• A Bayesian approach was used to model the functions represented by DNNs.
• Network architecture was varied by exploring transitions between ordered and chaotic regimes.
• The likelihood was approximated using the error spectrum of functions on data for Boolean function classification.
• Deep neural networks were trained using stochastic gradient descent.

Main Results:

• The Bayesian framework accurately predicted the posterior distribution of functions.
• The study demonstrated an accurate prediction for the posterior, measured for DNNs trained with stochastic gradient descent.
• The analysis revealed that structured data plays a critical role.

Conclusions:

• The success of DNNs is attributed to structured data and an Occam's razor-like inductive bias.
• This bias favors Kolmogorov simple functions, counteracting the exponential growth of function complexity.
• The findings highlight the importance of data structure and inherent biases in DNNs for effective supervised learning.