On the Universal Approximation Properties of Deep Neural Networks Using MAM Neurons.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Increasing neural network complexity leads to larger models, challenging deployment on resource-constrained devices.
• Multiply-and-ACcumulate (MAC) neurons are standard but memory-intensive.
• Multiply-And-Max/min (MAM) neurons offer selective behavior for pruning, promising significant memory footprint reduction.

Purpose of the Study:

• To theoretically characterize the capabilities of hybrid Multiply-And-Max/min (MAM) and Multiply-and-ACcumulate (MAC) neural networks.
• To validate the assumption that hybrid MAM&MAC architectures retain the expressive power of MAC-only networks.

Main Methods:

• Theoretical analysis of neural network architectures.
• Mathematical proofs demonstrating universal approximation properties.

Main Results:

• Two theorems are presented, proving the universal approximation capability of specific hybrid networks.
• Demonstrated that two hidden MAM layers followed by a MAC neuron (with optional normalization) form a universal approximator.

Conclusions:

• Hybrid MAM&MAC networks maintain the expressive power of traditional MAC-only networks.
• These findings support the use of MAM neurons for efficient neural network deployment on edge devices.
• The theoretical validation paves the way for practical implementation of memory-efficient deep learning models.