Towards understanding residual and dilated dense neural networks via convolutional sparse coding.

Area of Science:

• Deep Learning
• Computational Mathematics
• Computer Vision

Background:

• Convolutional Neural Networks (CNNs) achieve state-of-the-art results but lack theoretical understanding.
• Multilayer Convolutional Sparse Coding (ML-CSC) models offer a theoretical framework for plain networks.
• Factors like initialization and dictionary design impact ML-CSC performance.

Purpose of the Study:

• To propose novel multilayer models inspired by ResNet and MSDNet.
• To provide theoretical interpretations for ResNet's skip connections and MSDNet's dilated convolutions and dense connections.
• To enhance the performance and understanding of deep learning models through sparse coding.

Main Methods:

• Developed Residual Convolutional Sparse Coding (Res-CSC) and Mixed-Scale Dense Convolutional Sparse Coding (MSD-CSC) models.
• Derived skip connections as a special case of ML-CSC forward propagation.
• Analyzed Res-CSC and MSD-CSC to theoretically interpret dilated convolutions and dense connections.
• Implemented iterative soft thresholding algorithms for model optimization.

Main Results:

• Established a mathematical link between ResNet's skip connections and ML-CSC.
• Provided theoretical interpretations for dilated convolutions and dense connections within the ML-CSC framework.
• Demonstrated the effectiveness of Res-CSC and MSD-CSC through extensive numerical experiments.
• Showcased superior performance compared to existing methods.

Conclusions:

• Res-CSC and MSD-CSC offer significant theoretical insights into deep learning architectures.
• These models provide a clearer mathematical understanding of complex network components.
• The proposed models and their optimization methods are effective and competitive.