Nonlinear tensor train format for deep neural network compression.

Area of Science:

• Deep Learning
• Artificial Intelligence
• Computer Science

Background:

• Deep neural network (DNN) compression is crucial for deploying large models on resource-constrained devices.
• Tensor decomposition offers an efficient DNN compression strategy, but existing methods face accuracy loss or limited compression ratios.
• Current tensorization and mapping approaches for DNN compression have limitations, particularly for convolutional neural networks (CNNs).

Purpose of the Study:

• To propose a novel nonlinear tensor train (NTT) format for DNN compression.
• To address the accuracy loss associated with traditional tensor decomposition methods.
• To demonstrate the efficiency and effectiveness of NTT for both model compression and inference speed.

Main Methods:

• Researched various tensor decomposition techniques, focusing on Tensor Train (TT).
• Developed a novel Nonlinear Tensor Train (NTT) format by embedding nonlinear activation functions within TT contractions and convolutions.
• Applied NTT to compress DNNs, including weight matrices and convolutional kernels.

Main Results:

• NTT successfully compensates for accuracy loss typically observed with standard TT decomposition.
• Compressed DNNs using NTT maintain high accuracy on datasets like MNIST, UCF11, and CIFAR-10.
• Significant accuracy recovery was observed on large-scale datasets such as ImageNet, demonstrating NTT's scalability.

Conclusions:

• The proposed NTT format offers a viable solution for DNN compression, balancing model size, inference speed, and accuracy.
• NTT enhances the applicability of tensor decomposition for practical deep learning applications.
• This method provides a significant improvement over existing tensor train-based compression techniques.