LTNN: A Layerwise Tensorized Compression of Multilayer Neural Network

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Efficient deep learning models require memory-efficient neural network construction.
• Existing methods often struggle to balance compression rates with maintained accuracy.

Purpose of the Study:

• To introduce a novel layerwise tensorized formulation of a multilayer neural network (LTNN).
• To achieve significant neural network compression during training while maintaining accuracy.
• To improve inference speed compared to existing methods.

Main Methods:

• Reshaping multilayer neural network weight matrices into high-dimensional tensors with low-rank approximation.
• Developing a layerwise training method using a modified alternating least-squares algorithm with backward propagation for fine-tuning.
• Evaluating LTNN on standard benchmarks like MNIST and ImageNet12.

Main Results:

• LTNN achieved a 64x compression rate on the MNIST benchmark with no accuracy drop.
• On the ImageNet12 benchmark, LTNN demonstrated a 35.84x compression rate with approximately 2% accuracy loss.
• Inference speed was improved by 1.615x compared to existing works due to smaller tensor core ranks.

Conclusions:

• LTNN offers a highly effective approach for significant neural network compression.
• The method successfully balances compression with accuracy, making deep learning models more memory-efficient.
• LTNN presents a promising direction for developing faster and more compact deep learning models.