DRRNets: Dynamic Recurrent Routing via Low-Rank Regularization in Recurrent Neural Networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Recurrent Neural Networks (RNNs) excel at sequence learning but struggle with long sequences due to complex dependencies and gradient issues.
• Training RNNs for extended sequences presents challenges like vanishing/exploding gradients and high resource demands for deployment.

Purpose of the Study:

• To introduce Dynamic Recurrent Routing Neural Networks (DRRNets) to overcome limitations in training RNNs for long sequences.
• To enhance efficiency and performance in sequential learning tasks.

Main Methods:

• DRRNets dynamically allocate recurrent routes to shorten effective sequence lengths.
• Low-rank constraints are imposed on fully connected layers to significantly reduce model parameters.
• A novel optimization algorithm combining low-rank constraints and sparsity projection is developed for training.

Main Results:

• The proposed DRRNets demonstrate superior performance compared to existing methods in language modeling and speaker recognition tasks.
• Effectiveness verified across multiple popular sequential learning benchmarks.
• DRRNets successfully address challenges of long sequence training, gradient stability, and parameter efficiency.

Conclusions:

• DRRNets offer a significant advancement in training deep learning models for sequential data.
• The method provides a more efficient and effective approach to RNNs, particularly for long sequences.
• Dynamic routing and parameter reduction contribute to improved performance and deployability.