Exploring Structural Sparsity of Deep Networks Via Inverse Scale Spaces

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) achieve success through over-parameterization, but this leads to computational burdens.
• Training DNNs without over-parameterization risks getting trapped in local optima.
• Existing methods for network sparsity are often backward selection and computationally expensive.

Purpose of the Study:

• To address the need for systematic forward selection methods for learning structural sparsity in deep networks.
• To propose a novel approach for simultaneously exploring over-parameterized deep models and their structural sparsity.
• To develop an efficient method for growing deep networks with adaptive configurations.

Main Methods:

• A new approach based on differential inclusions of inverse scale spaces.
• A discretization scheme named Deep structure splitting Linearized Bregman Iteration (DessiLBI).
• Coupling a pair of parameters to generate models from simple to complex along dynamics.

Main Results:

• DessiLBI achieves comparable or better performance than existing optimizers in exploring sparse structures.
• The method identifies transferable "winning tickets" with comparable accuracy to fully trained models.
• Efficient network growth with adaptive filter configurations and reduced computational cost.

Conclusions:

• DessiLBI offers an effective and efficient solution for learning structural sparsity in deep networks.
• The method enables simultaneous exploration of over-parameterization and sparsity, leading to performant sparse models.
• DessiLBI facilitates efficient network growth and discovery of transferable sparse subnetworks.