State-Space Representations of Deep Neural Networks

Area of Science:

• Artificial Intelligence
• Dynamical Systems Theory
• Computational Neuroscience

Background:

• Neural networks are often analyzed as dynamical systems.
• Skip connections, like those in residual networks, are crucial for deep learning architectures.
• Understanding the underlying mathematical structure of these networks is key to improving their efficiency.

Purpose of the Study:

• To investigate the dynamical system properties of neural networks with skip connections.
• To derive closed-form solutions for state-space representations of specific network types.
• To analyze the impact of network order on state-space and embedding dimensions.

Main Methods:

• Modeling neural networks as dynamical systems governed by finite difference equations.
• Introducing higher-order dynamical equations through skip connections.
• Deriving closed-form solutions for additive dense and smooth networks.
• Analyzing state-space and embedding dimensions based on network order.

Main Results:

• Skip connections introduce Nth-order dynamical equations in layer-wise transformations.
• Closed-form solutions for state-space representations of Nth-order additive dense and smooth networks were found.
• Higher-order smoothness increases state-space dimension, impacting embedding capabilities.
• Parameter efficiency is improved by a factor of N compared to first-order residual networks.

Conclusions:

• The proposed framework provides an algebraic structure for deep neural networks.
• Higher-order network architectures offer significant parameter reduction for equivalent embedding dimensions.
• Numerical simulations validate the theoretical findings on benchmark datasets (CIFAR10, SVHN, MNIST).