Hybrid tensor decomposition in neural network compression.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Applied Mathematics

Background:

• Deep neural networks (DNNs) achieve state-of-the-art results but require significant computational resources.
• Increasing model complexity leads to growing storage consumption, hindering practical deployment.
• Tensor decomposition methods like tensor-train (TT) and tensor-ring (TR) offer potential for DNN compression.

Purpose of the Study:

• To investigate the effectiveness of the hierarchical Tucker (HT) tensor decomposition method for neural network compression.
• To compare the performance of HT and TT formats in compressing different components of convolutional neural networks (CNNs).
• To propose a hybrid tensor decomposition strategy for optimizing DNN compression.

Main Methods:

• Converted weight matrices and convolutional kernels of CNNs into both HT and TT tensor formats.
• Conducted comparative theoretical and experimental analyses of HT and TT compression effectiveness.
• Developed and evaluated a hybrid tensor decomposition strategy combining TT and HT formats.

Main Results:

• Hierarchical Tucker (HT) format demonstrated superior performance in compressing weight matrices.
• Tensor-train (TT) format proved more effective for compressing convolutional kernels.
• The proposed hybrid approach achieved better accuracy compared to using only TT or HT formats for CNN compression.

Conclusions:

• Hierarchical Tucker (HT) decomposition is a viable and effective method for compressing weight matrices in DNNs.
• A hybrid tensor decomposition strategy, leveraging the strengths of both TT and HT, offers a promising direction for efficient and accurate DNN compression.
• This research highlights the potential of hybrid tensor decomposition for advancing neural network compression techniques.