Compact Model Training by Low-Rank Projection With Energy Transfer

Area of Science:

• Deep Learning
• Machine Learning
• Computer Vision

Background:

• Low-rankness is crucial in traditional machine learning but underutilized in deep learning.
• Existing low-rank compression methods often suffer performance degradation due to suboptimal initialization.
• Low-rank methods receive less attention compared to pruning techniques in recent years.

Purpose of the Study:

• To develop a novel training methodology for low-rank compressed deep learning networks from scratch.
• To address the performance gap in low-rank compression by proposing an effective training strategy.
• To improve the efficiency and performance of deep learning models through advanced compression techniques.

Main Methods:

• Introduced Low-Rank Projection with Energy Transfer (LRPET) training method.
• Alternately applied stochastic gradient descent and projection onto low-rank manifolds.
• Implemented energy transfer to compensate for singular value reduction and mitigate gradient vanishing.
• Proposed Batch Normalization (BN) rectification to improve optimal low-rank approximation (LRA).

Main Results:

• LRPET achieves competitive performance by training compressed networks from scratch.
• The method demonstrates superior compression results compared to existing low-rank and state-of-the-art pruning methods on CIFAR-10 and ImageNet.
• Effective compression demonstrated in object detection, semantic segmentation, and Transformer-based models.
• Combined LRPET with quantization and hashing for enhanced compression.

Conclusions:

• LRPET offers a superior approach to low-rank network compression, outperforming previous methods.
• The proposed training strategy effectively preserves model capacity and performance.
• LRPET shows versatility and effectiveness across various deep learning architectures and tasks.