Compressing Deep Neural Networks With Sparse Matrix Factorization.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) are often overparameterized, containing a vast number of learnable parameters.
• Model compression is crucial for efficient deployment of DNNs, with sparsity being a key technique.
• Existing methods primarily focus on feedforward networks, leaving room for broader applicability.

Purpose of the Study:

• To present an alternative framework for learning sparse deep neural networks.
• To leverage matrix factorization for efficient DNN compression.
• To extend sparse learning techniques to recurrent networks.

Main Methods:

• Developed a framework that substitutes original parameter matrices with multiplications of highly sparse matrices.
• Established the theoretical underpinnings for this substitution.
• Empirically validated the method on various deep neural network architectures.

Main Results:

• The proposed method achieves substantial performance improvements over state-of-the-art compression techniques.
• Demonstrated effectiveness in compressing feedforward networks (MLPs, CNNs) and recurrent networks.
• Provided strong empirical evidence for the efficacy of matrix factorization in DNN sparsity learning.

Conclusions:

• Matrix factorization offers an effective approach for learning sparse deep neural networks.
• The method provides superior compression performance compared to existing techniques.
• The framework's applicability to recurrent networks broadens its practical relevance.