Neural Network Compression Based on Tensor Ring Decomposition.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) offer high accuracy but suffer from substantial memory and computational demands.
• These limitations hinder the deployment of DNNs on resource-constrained devices like desktops and mobile platforms.
• Low-rank factorization is a key technique for network compression, reducing model size by decomposing parameters.

Purpose of the Study:

• To explore the application of tensor ring (TR) factorization for compressing deep neural networks.
• To investigate the effects of parameter tensor reshaping and TR decomposition (TRD) on model compression.
• To develop an algorithm for optimizing tensor reshaping and TRD for maximal parameter reduction.

Main Methods:

• Proposed leveraging tensor ring (TR) factorization for neural network compression.
• Investigated the impact of parameter tensor reshaping and TR decomposition (TRD).
• Developed a prime factorization-based algorithm for optimal tensor reshaping and TRD.
• Introduced a novel tree structure and a top-to-bottom splitting algorithm to minimize computational complexity by optimizing core tensor execution order.

Main Results:

• The proposed algorithm achieves maximal parameter compression through optimal tensor reshaping and TRD.
• Different execution orders of core tensors significantly impact computational complexity.
• The developed scheduling algorithm effectively minimizes computational complexity.
• Extensive experiments on three neural network types and datasets demonstrated superior performance compared to state-of-the-art low-rank factorization methods.

Conclusions:

• Tensor ring factorization offers a powerful approach for compressing deep neural networks.
• The proposed prime factorization-based algorithm and tensor scheduling method significantly reduce memory consumption and computational complexity.
• This work enables more efficient deployment of DNNs on devices with limited resources.