Spectral pruning of fully connected layers.

Area of Science:

• Computational Neuroscience
• Machine Learning Theory

Background:

• Traditional neural network training optimizes individual weights.
• Understanding network structure and neuron importance is crucial for efficiency.

Purpose of the Study:

• To reformulate neural network training in spectral space.
• To leverage eigenvalues for ranking neuron importance and enabling network pruning.

Main Methods:

• Optimization of neural network eigenvalues and eigenvectors instead of individual weights.
• Ranking neurons based on associated eigenvalues.
• Developing pre- and post-processing pruning strategies.

Main Results:

• Demonstrated that eigenvalues can effectively rank node importance in neural networks.
• Achieved massively compacted networks through eigenvalue-based pruning.
• Maintained network performance despite significant reduction in neuron count.

Conclusions:

• Spectral space reformulation offers a novel approach to neural network optimization.
• Eigenvalue-based pruning is an effective strategy for creating efficient, high-performance neural networks.
• The proposed methods are versatile, applicable to diverse architectures and classification tasks.