Efficient shallow learning as an alternative to deep learning.

Area of Science:

• Computer Science
• Machine Learning
• Artificial Intelligence

Background:

• Complex classification tasks typically require deep learning (DL) architectures with many layers.
• Current DL models differ significantly from the human brain's structure.
• The conventional DL approach posits that early layers detect local patterns, with later layers identifying large-scale features.

Purpose of the Study:

• To investigate the relationship between layer depth, filter count, and error rates in generalized LeNet and VGG-16 architectures.
• To explore the potential of shallow learning models for complex classification tasks.
• To identify universal principles governing the performance of convolutional neural networks.

Main Methods:

• Analyzing generalized LeNet and VGG-16 architectures with a fixed ratio between the first and second convolutional layer depths.
• Applying power law analysis to model error rates as a function of the number of filters in the first convolutional layer.
• Investigating a conservation law related to convolutional layer size and depth.

Main Results:

• Error rates in generalized LeNet architectures decay as a power law with an increasing number of filters in the first convolutional layer.
• Extrapolation suggests shallow LeNet can achieve error rates comparable to deeper DL models on the CIFAR-10 database.
• A similar power law governs VGG-16, but with a higher computational cost.
• A conservation law (sqrt(layer size) * depth) was found to minimize error rates.

Conclusions:

• Shallow learning models, exemplified by LeNet, demonstrate efficient performance for complex tasks.
• The power law phenomenon suggests a universal behavior in convolutional neural network scaling.
• Further research and hardware development are recommended for efficient shallow learning implementation.