Parameter Efficient Neural Networks With Singular Value Decomposed Kernels.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Neural networks traditionally use matrix multiplications for connected layers.
• Representing weight matrices efficiently is crucial for deep learning models.

Purpose of the Study:

• To propose a new method for composing neural network weight matrices.
• To enhance parameter efficiency and performance of neural networks.
• To explore the optimization properties of the proposed method.

Main Methods:

• Weight matrices are decomposed into orthogonal basis matrices using the Kronecker product and Singular Value Decomposition (SVD).
• Orthogonal components are trained on the Stiefel manifold via the Cayley transform.
• Update equations for singular values and initialization routines are derived.
• Acceleration for stochastic gradient descent (SGD) optimization is discussed.

Main Results:

• The proposed method allows for more parameter-efficient weight matrix representations.
• Decomposed weight matrices achieve maximal performance in various neural architectures.
• The decomposed layers demonstrate reduced optimization dependency and improved conditioning.
• Training time may increase by up to a factor of 2.

Conclusions:

• The novel decomposition method offers significant advantages in parameter efficiency and model performance.
• The method's properties contribute to better-conditioned and less optimization-dependent neural network layers.
• The trade-off of increased training time is attributed to the optimization process on the manifold of orthogonal matrices.