Revisiting deep information propagation: Fractal frontier and finite-size effects

Area of Science:

• Deep Learning
• Computational Neuroscience
• Complex Systems

Background:

• Information propagation in deep neural networks (DNNs) is crucial for understanding how input correlations evolve across layers.
• Mean-field theory, commonly used, assumes infinitely wide networks, which is unrealistic for practical applications.
• Finite-size effects in neural network dynamics remain less understood.

Purpose of the Study:

• To investigate information propagation in randomly initialized, finite-width neural networks.
• To characterize the transition between ordered and chaotic regimes in these networks.
• To extend the analysis to convolutional neural networks (CNNs) and explore backward pass dynamics.

Main Methods:

• Analysis of information propagation in finite-size multilayer perceptrons (MLPs).
• Application of Fourier-based structured transforms for CNN analysis.
• Investigation of network dynamics independent of input data and optimization algorithms.

Main Results:

• The boundary between ordered and chaotic regimes in finite neural networks exhibits a fractal structure.
• This fractal behavior is observed in both MLPs and CNNs.
• Fractal patterns are also present in the backward pass (backpropagation) of finite-size networks.

Conclusions:

• Finite network width introduces fundamental complexity in neural network dynamics, characterized by fractal structures.
• The findings highlight the importance of finite network depth for balancing separation and robustness.
• The study reveals universal fractal dynamics in information propagation across different network architectures and pass directions.