On the complexity of neural network classifiers: a comparison between shallow and deep architectures

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Deep learning architectures with multiple hidden layers are increasingly favored for complex tasks.
• Existing theoretical results supporting the superiority of deep networks are limited.

Purpose of the Study:

• To investigate the impact of feedforward neural network depth on function complexity.
• To provide theoretical support for the effectiveness of deep architectures.

Main Methods:

• Introduced a novel complexity measure based on topological concepts for classification functions.
• Compared deep and shallow neural network architectures using sigmoidal activation functions.
• Derived upper and lower bounds on function complexity in relation to network depth.

Main Results:

• Deep networks were shown to implement functions of higher complexity compared to shallow networks.
• Function complexity was analyzed in terms of hidden units and activation functions.
• The findings support the hypothesis that deeper networks are more powerful.

Conclusions:

• Deep feedforward neural networks are theoretically capable of implementing more complex functions.
• Deeper architectures offer greater problem-solving capabilities with comparable resources.
• This research provides a theoretical foundation for the advantages of deep learning.