No Fine-Tuning, No Cry: Robust SVD for Compressing Deep Networks.

Area of Science:

• Machine Learning
• Computational Geometry
• Numerical Analysis

Background:

• Neural network compression is crucial for efficient deployment.
• Singular Value Decomposition (SVD) for k-rank approximation is common but requires fine-tuning.
• Fine-tuning demands significant computational resources, time, and access to the original training dataset.

Purpose of the Study:

• To develop a neural network compression algorithm that bypasses the fine-tuning step.
• To offer an alternative to SVD-based compression that is less sensitive to outliers.
• To provide a practical and provable approximation algorithm for L_p approximation in neural network compression.

Main Methods:

• Replaced the standard k-rank L2 approximation with L_p approximation (p>=1).
• Developed a novel approximation algorithm for L_p norm computation using computational geometry techniques.
• Evaluated the method on transformer-based models (BERT, DistilBERT, XLNet, RoBERTa) using the GLUE benchmark.

Main Results:

• The proposed L_p approximation method achieves compression without a fine-tuning stage.
• The algorithm demonstrates a theoretical advantage in handling outliers compared to L2 approximation.
• Experimental results validate the effectiveness of the L_p approximation approach on benchmark datasets.

Conclusions:

• The L_p approximation offers an efficient and resource-friendly alternative for neural network compression.
• This method eliminates the need for fine-tuning and original training data.
• The approach shows promise for broader adoption of compressed neural networks.