Power Law in Deep Neural Networks: Sparse Network Generation and Continual Learning With Preferential Attachment

Area of Science:

• Artificial Intelligence
• Network Science
• Computational Neuroscience

Background:

• Deep neural networks (DNNs) training demands significant computational resources.
• Existing DNNs suffer from inefficiencies in time and storage due to redundancy.
• Biological and social networks demonstrate high efficiency and scale-free properties.

Purpose of the Study:

• Investigate if DNN topology exhibits power law distribution, like biological networks.
• Explore utilizing power law topology for constructing efficient and compact DNNs.
• Analyze the correlation between network performance and power law distribution.

Main Methods:

• Modeled DNN connectivities using truncated power law distribution.
• Compared performance of various DNNs against power law distribution.
• Modeled preferential attachment in DNN evolution and continual learning.
• Proposed novel applications for sparse network generation and continual learning.

Main Results:

• Sparse DNN connectivities align with truncated power law distributions.
• High-performing DNNs exhibit strong correlation with power law distribution.
• Preferential attachment dynamics observed in DNNs during continual learning.
• Proposed applications demonstrated faster training, reduced storage, and improved sample efficiency.

Conclusions:

• Power law dynamics and preferential attachment are inherent in high-performing DNNs.
• Leveraging these dynamics enables the creation of accurate and compact DNNs.
• This research offers insights for designing more efficient deep learning architectures.