Deep Learning Model Compression With Rank Reduction in Tensor Decomposition.

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Deploying large neural network models on resource-constrained edge devices is challenging due to high bandwidth requirements.
• Existing model compression techniques often struggle to balance efficiency with accuracy preservation.

Purpose of the Study:

• To propose a novel deep learning model compression method.
• To enable efficient deployment of large neural networks on edge devices.
• To reduce communication overhead in federated learning applications.

Main Methods:

• A dual-model training strategy is employed.
• Iterative and adaptive rank reduction (RR) in tensor decomposition is utilized for regularization.
• Theoretical analysis of convergence and complexity is provided.

Main Results:

• The proposed method outperforms baseline compression techniques across various datasets (MNIST, CIFAR-10/100, ImageNet) and models (LeNet, VGG, ResNet, EfficientNet, RevCol).
• Achieved significant storage reduction (e.g., 10.41x for VGG-16) and speedup (e.g., 6.29x for VGG-16).
• Demonstrated substantial reduction in communication overhead (13.96x) in federated learning scenarios.

Conclusions:

• The novel DL compression method effectively reduces model size and computational requirements.
• The technique preserves model accuracy while significantly improving efficiency.
• Validated theoretical findings through extensive experiments, showing practical benefits for edge AI and federated learning.