Training of sparse and dense deep neural networks: Fewer parameters, same performance

Area of Science:

• Computational neuroscience
• Machine learning
• Artificial intelligence

Background:

• Deep neural networks (DNNs) can be trained in reciprocal space using transfer operators.
• Current spectral methods offer parameter compression but yield lower classification accuracy compared to direct space training.
• Existing methods face limitations in performance despite parameter space reduction.

Purpose of the Study:

• To propose a variant of spectral learning that enhances classification accuracy in DNNs.
• To reduce the computational cost associated with training deep neural networks.
• To achieve performance closer to conventional methods while maintaining parameter efficiency.

Main Methods:

• A novel spectral learning variant using two sets of eigenvalues for layer mappings.
• Tuning eigenvalues to control input node contribution and output node excitability (artificial homeostatic plasticity).
• Utilizing eigenvector matrix decomposition to bridge performance gaps.

Main Results:

• The proposed method significantly improves classification scores compared to previous spectral techniques.
• Trainable parameters scale linearly with network size, offering substantial computational savings.
• Performance closely approaches conventional direct space training, with reduced computational demands.
• The method facilitates the creation of sparse networks with high classification accuracy.

Conclusions:

• The enhanced spectral learning method provides a computationally efficient alternative to conventional DNN training.
• It achieves competitive classification performance while enabling significant parameter compression.
• The approach offers a promising direction for developing efficient and accurate deep learning models.