Learning Compressed Transforms with Low Displacement Rank.

Area of Science:

• Machine Learning
• Numerical Linear Algebra
• Deep Learning

Background:

• The low displacement rank (LDR) framework represents matrices using displacement operators and a low-rank residual.
• Current deep learning applications use LDR matrices with fixed operators, limiting their flexibility.
• This limits the ability to capture complex relationships and optimize model parameters effectively.

Purpose of the Study:

• To introduce a generalized class of LDR matrices with learnable displacement operators.
• To improve the efficiency and accuracy of deep neural networks through parameter compression.
• To provide theoretical bounds on the VC dimension for networks with these structured matrices.

Main Methods:

• Developed a novel class of LDR matrices with general, learnable displacement operators.
• Integrated these matrices into fully-connected, convolutional, and recurrent neural networks.
• Analyzed the VC dimension bounds for multi-layer neural networks utilizing these structured weight matrices.

Main Results:

• The proposed LDR matrices generalize existing constructions while maintaining compression and computational efficiency.
• Empirical results demonstrate reduced sample complexity for learning.
• Achieved superior accuracy compared to existing compression methods and even unstructured layers on image classification and language modeling tasks, using over 20X fewer parameters.

Conclusions:

• The generalized LDR framework offers a powerful and efficient method for deep learning model compression.
• Learnable displacement operators significantly enhance the performance and parameter efficiency of neural networks.
• This approach represents a promising direction for developing more scalable and effective deep learning models.