Recurrent neural network pruning using dynamical systems and iterative fine-tuning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Network pruning is crucial for reducing computational costs and improving inference speed in neural networks.
• Recurrent Neural Networks (RNNs) are powerful but computationally intensive, necessitating efficient pruning strategies.
• Existing pruning methods may not adequately preserve RNN performance during parameter reduction.

Purpose of the Study:

• To propose a novel RNN pruning method that models weight matrices as time-evolving signals using Linear Dynamical Systems (LDSs).
• To introduce a discrimination-aware L2 regularization for fine-tuning pruned RNNs.
• To develop an iterative fine-tuning approach using a larger model to guide the pruned model's optimization.

Main Methods:

• Representing RNN weight matrices as time-evolving signals and modeling them with Linear Dynamical Systems (LDSs).
• Pruning weight vectors exhibiting similar temporal dynamics.
• Applying a novel discrimination-aware L2 regularization during fine-tuning to penalize weights with minimal impact.
• Implementing an iterative fine-tuning strategy where a larger network guides a smaller pruned network.

Main Results:

• The proposed method achieved a significant improvement in perplexity by at least 0.62% on the PTB dataset.
• An improved F1-score of 1.39% was observed on the SQuAD dataset.
• The method outperformed state-of-the-art approaches, avoiding performance degradation often seen with other pruning techniques.

Conclusions:

• The novel RNN pruning method effectively reduces model size while enhancing performance.
• Modeling weights as LDSs and employing iterative fine-tuning are key to successful RNN compression.
• This approach offers a promising direction for developing efficient and high-performing RNNs.